method variance

1. Method Variance Spector (1987) Williams et al. (1989) Bagozzi et al. (1990) Tepper & Tepper (1993)

3. How big a problem is it? Common bias sources due to measurement method can be correlated to produce incorrect results May artificially inflate or suppress correlations between two variables I/O adage: �all self-report measures intercorrelate at .30�

4. Examples of Method Variance Acquiescence Tendency for respondent to agree with items regardless of content Most common in ambiguous test items and poorly designed tests Social Desirability Tendency to report socially desirable answers Is sometimes used as a personality construct in its own right

5. Spector(1987) Analyzed ten published studies with multi-method designs and strong correlations between methods for each construct Procedure (developed by Campbell and Fiske, 1959) Correlate between traits using same method Correlate between traits using different methods If there is method bias, first correlation will be significantly larger

6. Spector(1987) Did not find significant differences attributable to method variance Social desirability was also correlated and had minimal effect Acquiescence was examined by comparing responses to positively and negatively worded items. The effect was small to nonexistant

7. Williams et al. (1989) Limitations of the statistical analysis Spector used Inability to account for differential reliability Implicit assumptions assumption of uncorrelated and maximally dissimilar methods

8. Williams et al. (1989) Reanalysis of Spector�s data using confirmatory factor analysis (CFA) Tested five models with each data sample M1: null model (no correlation) M2: trait model (no method variance) M3: method model (no trait variance) M4: trait and method model with no correlation between method factors M5: trait and method model with correlated method factors

9. Williams et al. (1989) Results Comparing M1 to M3, M3 was significantly better based on chi-square difference, and accounted for a large amount of the correlation based on the normed-fit index (NFI) Comparing M2 to M4, M4 was significantly better in 9 studies based on chi-square difference, and accounted for a large amount of the correlation based on the NFI M1 better than M3 indicates a better fit to the null model by including method factors. M2 better than M4 indicates that adding method factors to the trait factors model results in a better fitM1 better than M3 indicates a better fit to the null model by including method factors. M2 better than M4 indicates that adding method factors to the trait factors model results in a better fit

10. Williams et al. (1989) Results (con�t) Comparing M4 to M5, M5 was significantly better in 7 studies based on chi-square difference, but did not account for much difference in fit M5 better than M4 indicates a better fit to the model by assuming that the method factors are correlated.M5 better than M4 indicates a better fit to the model by assuming that the method factors are correlated.

11. Williams et al. (1989) Conclusions Based on CFA, method factors play a large role in the constructs analyzed by Spector There is also evidence that the method factors are not independent, as required by the Campbell and Fiske analysis procedure

12. Bagozzi et al. (1990) Limitations of Williams et al. analysis Tests examined only overall effects of method factors No information for conclusions about individual tests Ignored other chi-square indicators for additional model information Did not include possibility of trait/model interaction

13. Bagozzi et al. (1990) Reanalysis of the Spector data Used traits-only model as a baseline against the traits and methods model 10 of 11 data sets fit the model for traits and methods Examined statistical significance of method factors on individual measures Five data sets had no significant method factor loads on any individual measures, 4 had significant effects on some measures, and 2 had significant effects on all measures 2 data sets had significant factors on all their measures, 1 data set had 7/8, two had about half significant, 1 data set had 1/10, and 5 had no significant individual factors.2 data sets had significant factors on all their measures, 1 data set had 7/8, two had about half significant, 1 data set had 1/10, and 5 had no significant individual factors.

14. Bagozzi et al. (1990) Used additional criteria to evaluate the trait and method model 9 of the 11 data sets fit the model without large unexplained correlations 2 of the data sets were not suited for the trait and method model Criteria of standardised residuals, (bottom of p553) Examination of parameters for improper estimates (often occur from model misspecification)Criteria of standardised residuals, (bottom of p553) Examination of parameters for improper estimates (often occur from model misspecification)

15. Bagozzi et al. (1990) Examined the data sets to see if an alternative model, the direct-product model, would fit better than the CFA One data set fit the alternate model, that did not fit the CFA model None of the other data sets fit the DPA model DPA is an alternative to the confirmatory factors model, which includes interactions between traits and methodsDPA is an alternative to the confirmatory factors model, which includes interactions between traits and methods

16. Bagozzi et al. (1990) Conclusions The CFA model, but not DPA, fits 9 of the data sets The DPA model, but not the CFA, fit one data set One data set did not fit either model Two data sets would have been miscategorized if only chi-squared goodness of fit tests were used

17. Tepper & Tepper (1993) Method variance within measures: Covariance among items on the same measurement due to item format Inflates interitem correlations Inflates reliability estimates

18. Tepper & Tepper (1993) Common measurement guidelines encourage practices that increase method variance within measures Simplifying questionnaires by using homogeneous item formats and scales Examining test-retest reliability by examining consistency in responses across repeated items within the test

19. Tepper & Tepper (1993) Scale development suggestions Heterogeneous item formats Placing dissimilar items together, rather than grouping items by content and format Random placement of dummy items that capture irrelevent content Skip back and forth in questionnaire to break up respondents� concentration Spread the questionnaire administration out over several sittings