Analyzing observed composite differences across groups: Is partial measurement invariance enough?

1. Analyzing observed composite differences across groups: Is partial measurement invariance enough? Holger Steinmetz Faculty of Economics and Business Administration Department of Human Resource Management, Small Business Enterprises, and Entrepreneurship University of Giessen / Germany

2. Introduction • Importance of analyses of mean differences For instance: • gender differences on wellbeing, self-esteem, abilities, behavior • differences between leaders and non-leaders on intelligence and personality traits • differences between cultural populations on psychological competencies, values, wellbeing • Usual procedure: t-test or ANOVA with observed composite scores • Latent means vs. observed means • Partial invariance as legitimation for the composite difference test • Research question: Effects of unequal intercepts and/or factor loadings across groups on composite differences

3. l1 d1 x1 l2 d2 x2 x l3 x3 d3 l4 x4 d4 Relationship between latent and observed means

4. xi l1 d1 x1 l2 d2 x2 x li l3 x3 d3 l4 ti x4 d4 x Relationship between latent and observed means

5. xi l1 d1 x1 l2 d2 x2 x li l3 x3 d3 l4 ti x4 d4 x Relationship between latent and observed means

6. x1 x2 x E(xi) x3 x4 k Relationship between latent and observed means xi l1 d1 l2 d2 li l3 d3 l4 ti d4 x

7. x1 x1 x2 x2 x x x3 x3 x4 x4 Group differences in intercepts and factor loadings Group A Group B xi E(xi) E(xi) x k

8. x1 x1 x2 x2 x x x3 x3 x4 x4 Group differences in intercepts and factor loadings Group A Group B xi E(xi) E(xi) x k

9. x1 x1 x2 x2 x x x3 x3 x4 x4 Group differences in intercepts and factor loadings Group A Group B xi E(xi) E(xi) x k

10. The study • Partial invariance: Some loadings / intercepts are allowed to differ • Research question: Is partial invariance enough for composite mean difference testing? • Pseudo-differences • Compensation effects • Procedure (Mplus): • Step 1: a) Specification of two-group population models with varying differences in latent mean, intercepts and loadings b) 1000 replications, raw data saved • Step 2: Creation of a composite score • Step 3: Analysis of composite differences • Step 4: Aggregation (-> sampling distribution)

11. The study Group B Group A x1 x1 k=.00 k=.00 k=.30 • Population model: • Two groups • One latent variable • Conditions: • 4 vs. 6 indicators • Latent mean difference: 0 vs. .30 • Intercepts: equal vs. one vs. two intercepts unequal in varying directions (-.30 vs. +.30) • Loadings: equal (l‘s = .80) vs. one vs. two loadings = .60 • Sample size: 2x100 vs. 2x300 • Dependent variables • Average composite mean difference • Percent of significant composite differences x2 x2 x x x3 x3 x4 x4 x5 x5 l=.80 l=.60 x6 x6 t=.00 t=-.30

12. Pseudo-DifferencesEffects on the average composite difference 0.30 0.25 1 intercept unequal 2 intercepts unequal 0.20 0.15 0.10 0.05 0.00 4 Ind. 6 Ind. 4 Ind. 6 Ind. N = 2 x 100 N = 2 x 300

13. Pseudo-DifferencesEffects on the probability of significant differences (Type I error) 0.60 All intercepts equal 0.50 1 intercept unequal 2 intercepts unequal 0.40 0.30 0.20 0.10 0.00 4 Ind. 6 Ind. N = 2 x 100

14. Pseudo-DifferencesEffects on the probability of significant differences (Type I error) 0.60 All intercepts equal 0.50 1 intercept unequal 2 intercepts unequal 0.40 0.30 0.20 0.10 0.00 4 Ind. 6 Ind. 4 Ind. 6 Ind. N = 2 x 100 N = 2 x 300

15. Compensation effectsEffects on the average composite differences 0.30 All intercepts equal 1 intercept unequal 2 intercepts unequal 0.25 Effect of unequal loadings 0.20 Effect of unequal intercepts 0.15 0.10 0.05 0.00 2 Loadings unequal Loadings equal 1 Loading unequal 4 Indicators

16. Compensation effectsEffects on the average composite differences 0.30 All intercepts equal 1 intercept unequal 2 intercepts unequal 0.25 0.20 0.15 0.10 0.05 0.00 2 Loadings unequal 2 Loadings unequal Loadings equal 1 Loading unequal Loadings equal 1 Loading unequal 4 Indicators 6 Indicators

17. Compensation effectsEffects on the probability of significant differences (Power) 0.90 All intercepts equal 1 intercept unequal 0.80 2 intercepts unequal 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 2 Loadings unequal 2 Loadings unequal Loadings equal 1 Loading unequal Loadings equal 1 Loading unequal N = 2x100 / 4 Indicators N = 2x300 / 6 Indicators

18. Summary • Pseudo-differences • Even one unequal intercept increases the risk to find composite differences • High sample size increases risk (up to 60% with two unequal intercepts) • Unequal factor loadings have only a low influence • Number of indicators reduces the risk – but not substantially • Compensation effects • Just one unequal intercept reduces the size of the composite difference to 50% • With a “small” sample size little chance to find a significant composite difference (power = .25 - .40) • Two unequal intercepts drastically reduce the composite difference: The power in the „best“ condition (2x300, 6 Ind.) is only .50

19. Conclusons • Most comparisons of means rely on traditional composite difference analysis • Researcher must not use supported partial invariance as a legitimation for using all items of the scale as a composite • Recommendations • Use SEM: • Testing latent mean differences under partial invariance possible • Greater power even in small samples - use only those items that were invariant in tests of invariance - Increasing number of items (will, however, probably violate the factor model)

20. Thank you very much! Contact: Holger.Steinmetz@web.de