240 likes | 1.23k Views
What is Method Variance?. Variance attributable to the measurement methodMethod Variance biases results when correlating different constructs that used the same measurement method. How big a problem is it?. Common bias sources due to measurement method can be correlated to produce incorrect resultsMay artificially inflate or suppress correlations between two variablesI/O adage:
E N D
1. Method Variance Spector (1987)
Williams et al. (1989)
Bagozzi et al. (1990)
Tepper & Tepper (1993)
3. How big a problem is it? Common bias sources due to measurement method can be correlated to produce incorrect results
May artificially inflate or suppress correlations between two variables
I/O adage: “all self-report measures intercorrelate at .30”
4. Examples of Method Variance Acquiescence
Tendency for respondent to agree with items regardless of content
Most common in ambiguous test items and poorly designed tests
Social Desirability
Tendency to report socially desirable answers
Is sometimes used as a personality construct in its own right
5. Spector(1987) Analyzed ten published studies with multi-method designs and strong correlations between methods for each construct
Procedure (developed by Campbell and Fiske, 1959)
Correlate between traits using same method
Correlate between traits using different methods
If there is method bias, first correlation will be significantly larger
6. Spector(1987) Did not find significant differences attributable to method variance
Social desirability was also correlated and had minimal effect
Acquiescence was examined by comparing responses to positively and negatively worded items. The effect was small to nonexistant
7. Williams et al. (1989) Limitations of the statistical analysis Spector used
Inability to account for differential reliability
Implicit assumptions
assumption of uncorrelated and maximally dissimilar methods
8. Williams et al. (1989) Reanalysis of Spector’s data using confirmatory factor analysis (CFA)
Tested five models with each data sample
M1: null model (no correlation)
M2: trait model (no method variance)
M3: method model (no trait variance)
M4: trait and method model with no correlation between method factors
M5: trait and method model with correlated method factors
9. Williams et al. (1989) Results
Comparing M1 to M3, M3 was significantly better based on chi-square difference, and accounted for a large amount of the correlation based on the normed-fit index (NFI)
Comparing M2 to M4, M4 was significantly better in 9 studies based on chi-square difference, and accounted for a large amount of the correlation based on the NFI M1 better than M3 indicates a better fit to the null model by including method factors.
M2 better than M4 indicates that adding method factors to the trait factors model results in a better fitM1 better than M3 indicates a better fit to the null model by including method factors.
M2 better than M4 indicates that adding method factors to the trait factors model results in a better fit
10. Williams et al. (1989) Results (con’t)
Comparing M4 to M5, M5 was significantly better in 7 studies based on chi-square difference, but did not account for much difference in fit M5 better than M4 indicates a better fit to the model by assuming that the method factors are correlated.M5 better than M4 indicates a better fit to the model by assuming that the method factors are correlated.
11. Williams et al. (1989) Conclusions
Based on CFA, method factors play a large role in the constructs analyzed by Spector
There is also evidence that the method factors are not independent, as required by the Campbell and Fiske analysis procedure
12. Bagozzi et al. (1990) Limitations of Williams et al. analysis
Tests examined only overall effects of method factors
No information for conclusions about individual tests
Ignored other chi-square indicators for additional model information
Did not include possibility of trait/model interaction
13. Bagozzi et al. (1990) Reanalysis of the Spector data
Used traits-only model as a baseline against the traits and methods model
10 of 11 data sets fit the model for traits and methods
Examined statistical significance of method factors on individual measures
Five data sets had no significant method factor loads on any individual measures, 4 had significant effects on some measures, and 2 had significant effects on all measures 2 data sets had significant factors on all their measures, 1 data set had 7/8, two had about half significant, 1 data set had 1/10, and 5 had no significant individual factors.2 data sets had significant factors on all their measures, 1 data set had 7/8, two had about half significant, 1 data set had 1/10, and 5 had no significant individual factors.
14. Bagozzi et al. (1990) Used additional criteria to evaluate the trait and method model
9 of the 11 data sets fit the model without large unexplained correlations
2 of the data sets were not suited for the trait and method model Criteria of standardised residuals, (bottom of p553)
Examination of parameters for improper estimates (often occur from model misspecification)Criteria of standardised residuals, (bottom of p553)
Examination of parameters for improper estimates (often occur from model misspecification)
15. Bagozzi et al. (1990) Examined the data sets to see if an alternative model, the direct-product model, would fit better than the CFA
One data set fit the alternate model, that did not fit the CFA model
None of the other data sets fit the DPA model DPA is an alternative to the confirmatory factors model, which includes interactions between traits and methodsDPA is an alternative to the confirmatory factors model, which includes interactions between traits and methods
16. Bagozzi et al. (1990) Conclusions
The CFA model, but not DPA, fits 9 of the data sets
The DPA model, but not the CFA, fit one data set
One data set did not fit either model
Two data sets would have been miscategorized if only chi-squared goodness of fit tests were used
17. Tepper & Tepper (1993) Method variance within measures: Covariance among items on the same measurement due to item format
Inflates interitem correlations
Inflates reliability estimates
18. Tepper & Tepper (1993) Common measurement guidelines encourage practices that increase method variance within measures
Simplifying questionnaires by using homogeneous item formats and scales
Examining test-retest reliability by examining consistency in responses across repeated items within the test
19. Tepper & Tepper (1993) Scale development suggestions
Heterogeneous item formats
Placing dissimilar items together, rather than grouping items by content and format
Random placement of dummy items that capture irrelevent content
Skip back and forth in questionnaire to break up respondents’ concentration
Spread the questionnaire administration out over several sittings