1 / 22

Testing factorial invariance in multilevel data: A Monte Carlo study

Testing factorial invariance in multilevel data: A Monte Carlo study . Eun Sook Kim Oi -man Kwok Myeongsun Yoon. The purpose of the present study. Investigate the influence of data dependency on the test of factorial invariance in multilevel data.

ezekiel-melton
Download Presentation

Testing factorial invariance in multilevel data: A Monte Carlo study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Testing factorial invariance in multilevel data: A Monte Carlo study EunSook Kim Oi-man Kwok Myeongsun Yoon

  2. The purpose of the present study • Investigate the influence of data dependency on the test of factorial invariance in multilevel data. • Comparisons between multilevel SEM (the model-based or the designed-based multiple-group multilevel CFA) and multiple-group CFA

  3. Measurement invariance studies in multilevel data • Multilevel SEM • The groups of interest are treated as random samples. • Good for measurement invariance testing in individual and organizational levels. • Multiple-group CFA • The number of groups is small or relatively finite (e.g., <20 or so). • Can be utilized to test measurement invariance over groups at the organizational level.

  4. Multilevel measurement model • A conventional measurement model • A multilevel measurement model

  5. In a multilevel measurement model The variance of the factor The variance of residual The variance of observed scores

  6. Examine measurement invariance in multilevel data The general multilevel CFA model

  7. Examine measurement invariance in multilevel data Study 1 Study 2 • Test at the between level • Test at the within level

  8. Data generation

  9. Design • Noninvariance in the organizational (study 1) and the individual level (study 2). • Manipulated factors • The number of clusters (CN):30, 50, 80 and 160 • Cluster size (CS): 10 and 20 • The size of ICC: .09, .20, and .33 • The total sample size was between 300 and 3,200, and 1,000 replications were in a condition.

  10. Data dependency on group membership: • Intraclass correlation (ICC)

  11. Analysis Tested a configural and a metric models. The LRT was used to test factorial invariance in multiple-group CFA . The Satorra-Bentler scaled chi-squared test was used in multiple-group multilevel CFA. Examined power and type I error. Conducted ANOVA to examine the influence of 3 manipulated factors on power and type I error.

  12. Results (Study 1) Low admissible rates.

  13. Analysis in study 2 Examined factorial invariance at the individual level using the design-based multilevel CFA, instead of the model-based one, due to the limitation of the SEM software.

  14. Results (study 2)

  15. Limitations and conclusions The explanations of the discrepant findings between study 1 and study 2 might be confounded by the type of analysis. It is necessary to take account of the data dependency in multilevel data. A substantially large number of clusters is required in model-based multiple-group multilevel modeling.

  16. Comments • Information of the accuracy of estimated parameters is not available. • MLR provides accurate factor loadings but underestimates the residual variances and standard errors at the between level. Having more clusters or higher ICC does not effectively compensate for this. • Groups of interest might occur at different levels simultaneously, and how to model them?

  17. Comments (Cont.) It is not clear how the model was constrained. This study seemed to constrain the variance of the latent factor to be equal across groups, which is not practical and reasonable in real data. The large effect size in the factorial invariance with a large sample size yielded very high power, and it is recommended reducing the effect size to medium.

  18. Comments (Cont.) Contrast to the findings of Jones-Farmer (2010), the inflated type I error occurred in CFA even though data dependency was considered small. Further examinations are needed to understand the controversial findings.

  19. Future studies • Further examinations in • Scalar and error variance models • Cross-level invariance • The issue of sample sizes using other estimation methods • In MLR, the min. sample size of the between level is 100. How about in FIML? • The assessment of model fit in ML SEM.

  20. Future studies (Cont.) Examine more than 1 items with different factor loadings. Implement purification in this study. Conduct equal means for the latent factor or equal intercepts of indicators to examine factorial invariance. Examine scalar invariance to see if the findings are similar to the findings of French and Finch (2010).

  21. Future studies (Cont.) The possibility to develop a new indicator in SEM, such as adding person fit (as IRT approaches do) to SEM approaches, instead of model fit only. Conduct the model-based multilevel CFA using other software to overcome the limitation in current SEM software.

More Related