Cultural differences as statistical artefacts?

1. Cultural differences as statistical artefacts? Reanalysing cross-national data with more advanced techniques Dr Michael Hoelscher Department of Education University of Oxford QMSS Conference Prague 21/06/2007

2. Overview • Context of the study • Introduction to data • Cultural differences within Europe – a first approach • Reanalysing the data with CFA • Applying a correction for measurement errors • Conclusions

3. 1. Context of the study • European integration and enlargement often discussed in economic terms only • However: Cultural influences might play a crucial role • Comparison of values in different spheres for all countries in the EU (Religion, Family and Gender, Economy, Welfare State, Democracy) • “Normative” starting point: Position of the EU institutions, as found in its body of law and the treaties • Three year project, financed by VolkswagenStiftung

4. 2. Introduction to data • European Values Study • 1999/2000 • Wide variety of topics • Including all member and applicant countries of the EU (except Cyprus) • 28 countries are compared in our study • Today the focus is on “Democracy and Civic Society” • Secondary analysis • Indicators are not always “perfect”

5. 2. Introduction to data Democracy: • 4 Indicators • “Having a strong leader” (v216) • “Having the army ruling” (v218) • “Having a democratic political system” (v219) • “Democracy may have problems, but best form of government” (v220) (all measured on a scale with 4 categories)

6. 2. Introduction to data Civic Society: • 2 Indicators • “People can be trusted” (v66) • “Membership in voluntary organisations” (Index of membership in 14 groups; trade union membership is ignored)

7. 3. Cultural differences within Europe – a first approach Methods • Comparisons of raw country means for each indicator • Integration of single indicators by using a discriminant analysis (see Fuchs/Klingemann 2002 in “West European Politics”) • Explanation of differences on the individual level by multiple regressions

8. 3. Cultural differences within Europe – a first approach Results • Large differences between the countries, but also within the countries • Old member countries support position of EU most, followed by new members • Bulgaria, and especially Romania and Turkey showed much lower support

9. 3. Cultural differences – a first approach Overall support for the EU’s position in the field of Democracy/Civic Society (RANK)

10. 4. Re-analysing the data with CFA • Aim • To compare two different methods • Not: Building the best model! • Balance of model fit and equivalence of approaches is needed

11. 4. Re-analysing the data with CFA Advantages of CFA • Generally • CFA is the more appropriate technique • More flexible • Can easily be extended to an explanatory SEM

12. 4. Re-analysing the data with CFA Advantages • Measurement model • Test if measurement is the same in different countries and therefore a comparison is appropriate • Correction for measurement error possible (Saris/Gallhofer 2007)

13. 4. Re-analysing the data with CFA Advantages • Structural model • Relationship between “democracy” and “civic society” can be estimated

14. 4. Re-analysing the data with CFA Great Britain, N = 728

15. 4. Re-analysing the data with CFA Running the model for all 25 countries without constraints • Chi-square = 333.29, df = 175, p-value=.000 • CFI = .988 • RMSEA = .006 (adjusted: 0.032) All modification indices within the countries are well below 20, in most cases below 5 • One can assume configural invariance

16. 4. Re-analysing the data with CFA 1. Model “Unconstrained” Chi-square= 333,29 df = 175 CFI = .988 RMSEA = .006 (adjusted .032) Introducing constraints: Model comparison 2. Model “Equal Measurement Weights” Chi-square= 725,5 df = 271 CFI = .965 RMSEA = .008 (adjusted .04) 3. Model “Equal Measurement Weights and Intercepts” Chi-square= 4752.256 df = 367 CFI = .666 RMSEA = .023 (adjusted .115)

17. 4. Re-analysing the data with CFA 1. Model “Unconstrained” => Configural invariance can be assumed Introducing constraints: Model comparison 2. Model “Equal Measurement Weights” => Metric invariance can be assumed 3. Model “Equal Measurement Weights and Intercepts” => Scalar invariance can not be assumed! => Mean comparison is (in priniciple) not appropriate with this model => Adjustments (freeing some parameters)

18. 4. Re-analysing the data with CFA Comparison of ranks

19. 5. Correction for measurement errors • SEM allows to correct for measurement errors • Saris, Gallhofer et al. (2007) have introduced a tool to estimate the quality (reliability and validity) of an instrument • From a huge amount of MTMM experiments they estimated the influence of certain characteristics on the quality • By coding one’s own questions one can predict their quality => http://www.sqp.nl/

20. 5. Correction for measurement errors Idea: • What has to be equal for cross-country-comparisons is the factor structure • The quality of the instrument might influence this factor structure, • if one does not correct for measurement error • if the quality is different in different countries • “We suggest that equivalence should (…) be tested by the equality of loadings based on the observed covariance matrix corrected for measurement error”

21. 5. Correction for measurement errors Indicators True scores Latent concept (by definition) T1 q1 y1 e1 λ1 F λ2 T2 q2 y2 e2 T3 λ3 q3 y3 e3

22. 5. Correction for measurement errors Applying the correction to a subsample of 9 countries: • “Democracy”-indicators • The validity was nearly 1 for all countries • Reliability is different in countries, but reasonably good • Problems with the “Civic Society”-indicators • Unable to code the quality of the index straightforward • Low quality of the “Trust” variable

23. 5. Correction for measurement errors Results: • Factor loadings increase • Model fit decreases very slightly • At least for this specific subsample the ranks do not change • Check for whole sample, especially the “difficult” cases, is still missing

24. 6. Conclusions • Advantages of the SEM approach • More appropriate • More flexible (integration of additional indicators) • Can detect problems with measurement model • Easily extendable to an explanatory model • Relationship between the latent constructs can be estimated

25. 6. Conclusions • “Problems” of the SEM approach • More demanding (data quality) • Is it realistic to assume equal means and factor loadings over so many countries? • Partial invariance? • Taking requirements very seriously wouldn’t allow a comparison of all countries

26. 6. Conclusions • Comparing the “outcome” of the three methods: • Small differences for the overall ranking • The methods seem to come to pretty similar results • However: Some extreme cases (Turkey), couldn’t be included or shifted quite a lot (Romania)

27. Thank you! Quantitative Methods in the Social Sciences Conference, Prague, 20-23 June 2007 Dr Michael Hoelscher Department of Education University of Oxford michael.holscher@edstud.ox.ac.uk

28. Literature • Michael Hoelscher (2006): Wirtschaftskulturen in der erweiterten EU. Die Einstellungen der Buergerinnen und Buerger im europaeischen Vergleich. Wiesbaden: VS Verlag • Juergen Gerhards (unter Mitarbeit von Michael Hoelscher) (2005, second edition 2006): Kulturelle Unterschiede in der Europaeischen Union. Wiesbaden: VS Verlag • Dieter Fuchs/Hans-Dieter Klingemann (2002): Eastward Enlargement of the European Union and the Identity of Europe. West European Politics, 25, 2: 19-54. • Willem E. Saris/Irmtraud Gallhofer (2007): Design, Evaluation, and Analysis of Questionnaires for Survey Research. Wiley.