How to evaluate the cross-cultural equivalence of single items

1. How to evaluate the cross-cultural equivalence of single items Melanie Revilla, Willem Saris RECSM, UPF Zurich – 15/16 July

2. Cross-cultural equivalence • Usually • Discussed in the frame of cross-national research • Idea: in different countries people can express themselves in different ways • Different cultures can also be defined on other criteria (e.g. language) • Procedure: can be applied in similar way to all kinds of different groups • “Equivalence”  measurement equivalence • 2 persons with the same opinion will give the same answer (whatever their group) • Important • because observed differences may result from non equivalent measures and not be real differences • If measurement equivalence does not hold • cannot make comparison across groups!!

3. Important distinction (Northrop, 1947) • Concept by Intuition (CI) • Simple concepts that can be measured directly • Single item • Ex: trust in the parliament • Concept by Postulation (CP) • Complex concepts that cannot be measured directly • Also called “construct” • Need several CI to measure them • Ex: political trust: trust in parliament + legal system + police … • Classic procedure to test for equivalence for CP but not for CI  start with a reminder of the procedure for CP

4. When we have multiple indicators CP / complex concepts

5. Basic Confirmatory Factor Analysis model τ1 Y1 Answer Trust in the parliament λ11 e1 CP1 λ21 Y2 τ2 Answer Trust in the legal system e2 λ31 Political trust τ3 Y3 Answer Trust in the police e3 intercepts slopes error terms [ ≈ regression equation Yi = τi + λij CP1 + ei i = 1,2,3 Independent variable Dependent variable

6. Multiple Group CFA approach • Multiple group: • possible to test for equality of the parameters in the different groups • constraints across groups • Can be extended to more groups Group 1 Group 2 ?

7. Different levels of invariance (Meredith, 1993) Group 1 • Configural • Same model holds in all groups • Metric • Configural + Slopes (λij) the same in all groups • Sufficient for comparison of relationships • Scalar • Metric + Intercepts (τi) the same in all groups • Sufficient for comparison of means • More: error terms, etc… ? Group 2 =

8. In practice • Analyses can be done with standard SEM softwares • LISREL/Mplus • based on covariance matrices & means • recommended sample size: >200 in each group • 3-step procedure: configural, metric, scalar • syntax quite easy to get estimates • More tricky but crucial step: testing

9. Testing the model • Assessing global fit • Chi2 test / Fit indices: RMSEA (<.05), CFI (>.9), etc… • Limits: Depends on sample size / Sensitive to deviations from normality • Assessing local fit • Saris & Satorra  should test at the parameter level + take into account type II errors (H0 not rejected despite being false) • JRule software (van der Veld, Saris, Satorra) + Jrule for Mplus (Oberski) • See next presentation! • Always check if estimates are really different • Difference may be statistically significant but not substantially meaningful • Partial invariance • What if some indicators are equivalent but not all? • Consistent estimates of the means of the latent variables if at least 2 indicators are scalar invariant(Byrne, Shavelson, Muthén, 1989)

10. If we have single indicator CI / simple concepts

11. Single items Testing equivalence single items Testing equivalence for CI Group 1 Group 2 ? CI1 τ1 CI1 τ1 Y1 Y1 λ11 λ11 e1 e1 Yi = τi + λij CIi + ei

12. Single  multiple indicators? • Problem: model just presented not identified • “Single indicator” = single trait in fact • But possible to use multiple methods • So for CI: • Only one trait, but we can always have more than one method • Several indicators = same trait asked using different methods

13. Can apply again MGCFA Similar at the previous model (for CP) but now different methods instead of different traits measuring a same concept Group1 Group 2 ? 11 points 6 points Trust in the parliament 4 points

14. Same procedure • Different levels of invariance as for CP • Configural • Metric • Scalar • etc • Same procedure to get the estimates and test the model • Multiple group analyses • Test of the model: global / local fit • Partial equivalence

15. Problem for the CI • Fix the scale? • As before, necessary to fix the scale of the LV • Usually, fix the first loading to 1 • Can be done here too • Other loadings are relative to the first one • But need to be done in all groups • If there are differences for the method whose loading is fixed to 1 across groups, may be problematic • Should try to use methods that have been shown to be the most similar across groups: e.g. fixed reference points

16. General model τ11 Y11 e11 1 u1 α1 CI1 τ21 v21 Y21 e21 v31 c1 Y31 τ31 e31 Y12 τ12 1 u2 e12 CP1 α2 c2 v22 CI2 Y22 τ22 e22 v32 Y32 τ32 e32 c3 u3 Y13 τ13 1 α3 e13 CI3 v23 Y23 τ23 e23 v33 Y33 τ33 e33 • Even when working on CP: better to use different methods

17. Final remarks / CCL

18. Equivalence single items • Need to repeat the same item with different methods • 3 or more repetitions • Multi Methods (MM)? • Same persons get the question several times using different methods • Limit: 20 minutes at least to avoid memory effects (Van Meurs & Saris, 1990) • Mix with Split-Ballot (SB) design? • Random assignment of respondents to different versions of the questionnaire • “SB-MM” (CI) or SB-MTMM (CP)?

19. Conclusion Measurement equivalence can be assessed both for CP and CI using Multiple Group Confirmatory Factor Analysis For CP, process already well-known and used a lot For CI, possible do similar analyses  But necessary to repeat questions!!  specific data So testing equivalence of simple item can be done using a (SB)-(MT)MM approach Similar to what exists in the ESS for CP: main + supplementary questionnaires (different versions) With extension for concepts by intuition

20. In summary • To test single item equivalence • Use multiple methods • Do everything as for multiple items equivalence

21. Thank you for your attention! Questions?