Alternate Approaches to Modeling MTMM Data in Organizational Research

1. Alternate Approaches to Modeling MTMM Data in Organizational Research David J. Woehr Department of Management University of North Carolina Charlotte

2. Agenda… • Brief overview/review of MTMM matrix method • Overview of alternate analytic approaches • Factor analytic approach • ANOVA-based approach • Highlight the linkage between the two approaches • Present and demonstrate a combined approach

3. Acknowledgements… • Woehr, D.J., Putka, D.J., & Bowler, M.C. An examination of G-theory methods for modeling multitrait-multimethod data: Clarifying links to construct validity and confirmatory factor analysis. Organizational Research Methods, 15(1), 134-161.

4. Multitrait-Multimethod (MTMM) approach originates with: • Campbell, D.T, & Fiske, D. W. (1959). Convergent and discriminant validation by the Multitrait-Multimethod Matrix. Psychological Bulletin, 56(2), 81–105. • Now classic article: • 6,045 citations (Web of Science) • 12,233citations (Google Scholar)

5. Campbell & Fiske (1959) emphasize 3 aspects of validation process… • ‘Trait-method unit’ • Every measure is a combination of specific trait (construct) and measurement method; • Two types of validity evidence • Convergence • Different methods of assessing a given construct should converge; • Discrimination • Measures of different traits should differ from measures of other traits.

6. Campbell & Fiske (1959) emphasize 3 aspects of validation process… • Method variance • Requires at least 2 traits measured with 2 method

7. MTMM Correlation Matrix…

8. MTMM Correlation Matrix… Monotrait-Monomethod r’s ‘Reliability Diagonals’

9. MTMM Correlation Matrix… Monotrait-Heteromethod (MTHM) r’s ‘Validity diagonals’

10. MTMM Correlation Matrix… Heterotrait-Monomethod (HTMM) r’s Discrimination

11. MTMM Correlation Matrix… Heterotrait-Heteromethod (HTHM) r’s Discrimination

12. Campbell & Fiske (1959) specify several decision criteria…

14. Campbell & Fiske (1959) highlight…. • Distinction between reliability and validity is not always clear • Method similarity is a matter of degree • Distinction between ‘trait’ and ‘method’ not always clear

15. Distinction between reliability and validity is not always clear… • Both reliability and validity require agreement between measures • Reliability is agreement between two measures of the same trait via maximally similar methods, • Validity is agreement between two measures of the same trait via maximally different methods. • Method independence is an key assumption of the Campbell & Fiske (1959) approach

16. Method similarity is a matter of degree… • “The interpretation of the validity diagonal in an absolute fashion requires the fortunate coincidence of both an independence of traits and an independence of methods, represented by zero values in the heterotrait-heteromethod triangles.” C&F (1959) p. 84.

17. MTMM Correlation Matrix…

18. Method similarity is a matter of degree… • “Independence is, of course, a matter of degree, and in this sense, reliability and validity can be seen as regions on a continuum…” C&F (1959) p. 83.

19. Distinction between ‘trait’ and ‘method’ not always clear… • Trait is systematic construct-relevant variance; method is systematic construct-irrelevant variance • “The distinction between trait and method is of course relative to the test constructor’s intent. What is an unwanted response set for one tester may be a trait for another…” C&F (1959) p. 85

20. Two primary issues emerge… • Validity criteria are qualitative and often difficult to interpret • i.e., ambiguity with respect to what constitutes satisfactory results • Results are generally discouraging • “Matrices published today continue to be about as unsatisfactory as those published 33 years ago. … We have yet to see a really good matrix: one that is based on fairly similar concepts and plausibly independent methods and shows high convergent and discriminant validation by all standards. Is such a matrix possible?”Fiske & Campbell (1992) p. 393.

21. Formal statistical techniques… • Two ‘traditions’/approaches emerge: • The factor analytic approach • EFA • CFA • Largely the method of choice for analyzing MTMM data • The ANOVA approach • G-theory • Other • Nonparametric approaches • Generalized Proximity Function (Hubert & Baker, 1978;1979) • Smallest Space Analysis (Levin, Montag,& Comrey, 1983) • Nonparametric MDS

22. Primary goal of this presentation… • Highlight the linkage between the two approaches • Demonstrate the relations between g-theory variance components and MTMM correlations • Link variance components to convergent-discriminant validity ideas • Describe the structural model implied by g-theory models • Show how CFA may be used to estimate g-theory parameters • Provide a brief empirical example

23. CFA Model… • Xtm = αtxTt+ βmxMm+ Εtm • Observed score for a given measure of a trait (T) with method (M) • Marsh & Grayson (1995) • Provide review of variations of CFA models with multiple dimension/method factors

24. Uncorrelated Traits & Methods Model… t1m1 Trait 1 t2m1 Trait 2 t3m1 t1m2 Trait 3 t2m2 Method 1 t3m2 t1m3 Method 2 t2m3 Method 3 t3m3

25. Complete Model… t1m1 Trait 1 t2m1 Trait 2 t3m1 t1m2 Trait 3 t2m2 Method 1 t3m2 t1m3 Method 2 t2m3 Method 3 t3m3

26. Correlated Traits, Correlated Methods (CTCM) Model … t1m1 Trait 1 t2m1 Trait 2 t3m1 t1m2 Trait 3 t2m2 Method 1 t3m2 t1m3 Method 2 t2m3 Method 3 t3m3

27. Correlated Trait, Uncorrelated Method (CTCU) Model … t1m1 Trait 1 t2m1 Trait 2 t3m1 t1m2 Trait 3 t2m2 Method 1 t3m2 t1m3 Method 2 t2m3 Method 3 t3m3

28. Problems with the CFA approach… • Analyses often result in nonconvergence and/or improper estimates • Heywood cases, negative variances, and out-of-range factor correlations • Marsh & Bailey (1991) • Analyzed 435 matrices – 77% failed to converge on a proper solution • Lance, Woehr, & Meade (2007) • Generated 500 matrices corresponding to each of 3 ‘true’ models • CFA’s converged to an admissible solution for only 57% of the matrices • Of these, fit indices indicated a high level of fit regardless of whether or not estimated model matched the population model

29. Problems with the CFA approach… • “Campbell and Fiske (1959) proposed the use of the MTMM matrix to assess convergent and discriminant validity. Some 30 year later, exactly how to evaluate statistically these two validities is not known.” Kenny & Kashy (1992) • Kenny & Kashy recommend using a CTCU model

30. Correlated Trait, Correlated Uniqueness (CTCU) Model … t1m1 Trait 1 t2m1 t3m1 t1m2 Trait 2 t2m2 t3m2 t1m3 Trait 3 t2m3 t3m3

31. Problems with the CTCU model… • Method non-independence bias • To the extent that method factors are not independent, trait factor loadings will be upwardly biased (e.g., Lance, Nobel, & Scullen, 2002)

32. Other models also proposed… • Simplified model • e.g., The Correlated-Trait, Correlated Method (Minus One) Model • Eid (200); Eid et al. (2003)

33. Correlated Trait, Correlated Method Minus One (CTC(M-1)) Model… t1m1 Trait 1 t2m1 Trait 2 t3m1 t1m2 Trait 3 t2m2 Method 1 t3m2 t1m3 Method 2 t2m3 t3m3

34. Other models also proposed… • Multiplicative models • e.g., Direct Product Model • Swain (1975); Browne (1984)

35. Alternate analytic ‘tradition’/approach…. • ANOVA based approach • Generalizability theory (G-Theory) • Not widely used in organizational research • More popular in educational research

36. Alternate analytic ‘tradition’/approach…. • Contributing problems: • Not as flexible as CFA approach • More restrictive assumptions • No clear framework for linking G-theory outcomes to traditional MTMM idea of construct validity • Usually considered in the context of reliability • High level of jargon and perceived complexity of estimating variance components

37. G-theory model… • Based on random effects ANOVA model • P x T x M • Person (P) crossed with trait (T) crossed with method (M) • Each observed score is an additive function: • Xptm = μ + ηp + ηt + ηm + ηpt + ηpm + ηtm + ηptm,r • Each term has a corresponding variance component • σ2expected total = σ2p + σ2t+ σ2m+ σ2pt + σ2pm + σ2tm + σ2ptm,r • σ2expected observed = σ2p + σ2pt + σ2pm + σ2ptm,r.

38. G-theory model… • Assumptions • All models effects are: • Independent • Identically distributed • Means = 0 • Specific variance component associated with each effect • Variance components: • Focus of estimation • Can be estimated directly • GENOVA • SPSS VARCOMP • SAS PROC MIXED

39. Relations between variance components and MTMM correlations… • Can directly translate variance components into MTMM r’s: • AverageMTHM r = σ2p + σ2pt, • AverageHTMM r = σ2p + σ2pm, • AverageHTHM r = σ2p

40. Relations between variance components and MTMM correlations… • Can directly translate variance components into MTMM r’s: • AverageMTHM r = σ2p + σ2pt, • AverageHTMM r = σ2p + σ2pm, • AverageHTHM r = σ2p • And vice versa: • σ2p = Average HTHM r • σ2pt = Average MTHM r - Average HTHM r • σ2pm = Average HTMM r - Average HTHM r • σ2ptm, r = 1 - s2p + s2pt + s2pm

41. MTMM Correlation Matrix…

42. Relations between variance components and MTMM correlations… • Allows for establishing a set of indices corresponding to Campbell & Fisk’s original criteria

44. Univariate G-theory Analogues of Campbell and Fiske (1959) Definitions of Convergence, Discrimination, and Method Effects Note. The interpretations above assume one is dealing with variance components standardized against observed variance (i.e., s2p + s2pt + s2pm + s2ptm,r = 1).

45. Univariate G-theory Analogues of Campbell and Fiske (1959) Definitions of Convergence, Discrimination, and Method Effects Note. The interpretations above assume one is dealing with variance components standardized against observed variance (i.e., s2p + s2pt + s2pm + s2ptm,r = 1).

46. Univariate G-theory Analogues of Campbell and Fiske (1959) Definitions of Convergence, Discrimination, and Method Effects Note. The interpretations above assume one is dealing with variance components standardized against observed variance (i.e., s2p + s2pt + s2pm + s2ptm,r = 1).

47. Univariate G-theory Analogues of Campbell and Fiske (1959) Definitions of Convergence, Discrimination, and Method Effects Note. The interpretations above assume one is dealing with variance components standardized against observed variance (i.e., s2p + s2pt + s2pm + s2ptm,r = 1).

48. Univariate G-theory Analogues of Campbell and Fiske (1959) Definitions of Convergence, Discrimination, and Method Effects Note. The interpretations above assume one is dealing with variance components standardized against observed variance (i.e., s2p + s2pt + s2pm + s2ptm,r = 1).

49. Using CFA to estimate variance components… • Univariate g-theory implies a very constrained factor model • Equal trait factor variances • Equal method factor variances • Equal unique variances • Factor correlations are zero • Error covariances are zero

50. Structural Model Implied by Univariate G-Theory Model of MTMM Data… t1m1 Trait 1 t2m1 σ2p Trait 2 t3m1 t1m2 σ2pt Person Trait 3 t2m2 Method 1 t3m2 σ2pm t1m3 Method 2 t2m3 σ2ptm,r Method 3 t3m3