1 / 28

Professor Jane Zavisca University of Arizona April 12, 2013 janez@u.arizona

SSRC Eurasia Quantitative Methods Webinar Cultural Context and Measurement Validity in Comparative Survey Research. Professor Jane Zavisca University of Arizona April 12, 2013 janez@u.arizona.edu. Core dilemmas.

keegan-rice
Download Presentation

Professor Jane Zavisca University of Arizona April 12, 2013 janez@u.arizona

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SSRC Eurasia Quantitative Methods WebinarCultural Context and Measurement Validity in Comparative Survey Research Professor Jane Zavisca University of Arizona April 12, 2013 janez@u.arizona.edu

  2. Core dilemmas • In comparing groups, want to be sure observed differences (or similarities) are substantive, not artifactual • Simultaneous need for: • Identities: universal measures with comparable meanings across contexts • Equivalents: Particular measures that capture same concepts across contexts • In cross-national surveys, often have identities – but are they equivalent? Bollen, Kenneth A., Barbara Entwisle, and Arthur S. Alderson. 1993. “Macrocomparative Research Methods.” Annual Review of Sociology 19 (January 1): 321–351. doi:10.2307/2083391. Przeworski, Adam, and Henry Teune. 1966. “Equivalence in Cross-National Research.” Public Opinion Quarterly 30 (4) (December 21): 551–568. doi:10.1086/267455. Heath, Anthony, Stephen Fisher, and Shawna Smith. 2005. “The Globalization of Public Opinion Research.” Annual Review of Political Science 8 (1): 297–333. doi:10.1146/annurev.polisci.8.090203.103000.

  3. Measurement equivalence • Functional equivalence: concordance of meaning, of constructs as well as questions • Measurement invariance: formal statistical equality of parameters in measurement models

  4. Sources of non-equivalence • Errors of observation • Substantive differences in meaning: linguistic, contextual • Systematic differences in response styles: acquiescence bias, extreme response bias, social acceptability bias • Errors of non-observation • non-response bias • Sampling approach Heath, Anthony, Jean Martin, and TheesSpreckelsen. 2009. “Cross-national Comparability of Survey Attitude Measures.” International Journal of Public Opinion Research 21(3): 293–315. doi:10.1093/ijpor/edp034.

  5. Language example: happiness • Poor linguistic equivalence • English: “Are you a happy person?” • Russian: ”Вы счастливый человек?/ Вы счастливчик?» • Could be interpreted as “Are you a lucky person?” • О счастливчик! = name of “Who Wants to Be a Millionaire” reality show in Russian • Better linguistic equivalence • English: “Are you:very happy, pretty happy, not too happy, not happy at all.” • Russian: Вы: очень счастливы, довольно счастливы, не очень счастливы, очень несчастны.» See: RUSSET Panel Survey: www.vanderveld.nl World Database of Happiness: http://www1.eur.nl/fsw/happiness/index.html

  6. Relationship between happiness & life satisfaction • “Varies with the cultural and linguistic environment in which it is studied.” • Constructs vs. questions: differences in meaning even with linguistic equivalence • In Russian surveys • Low correlations between general happiness and life satisfaction • High correlation between general happiness and satisfaction with personal relationships; • high correlation between life satisfaction and satisfaction with finances. Saris, Willem E., and Anna Andreenkova. 2001. “Following Changes in Living Conditions and Happiness in Post Communist Russia: The RUSSET Panel.” Journal of Happiness Studies 2: 95-109.

  7. Contextual difference: homeownership • Conventional definition of homeowner: resident of “owner-occcupied household” • Russia has highest rate of young homeownership (ages 21-35) in Western & Eastern Europe according to this definition (about 85%). • But most are not living autonomously

  8. “Homeownership” Rates Ages 21-35, with and without extended family

  9. DETECTING MEASUREMENT NON-EQUIVALENCE USING MULTIPLE INDICATORS AND LATENT VARIABLE MODELS

  10. Limitations of single measures • Impossible to statistically test whether observed difference (or similarity) is meaningful • Example: consumer ethnocentrism • Russians are more likely than Canadians to agree with the statement: “There should be very little trading or purchasing of goods from other countries unless out of necessity.” • Possible sources of non-equivalence • Translation • Different interpretation, measuring different constructs Saffu, Kojo, and John Hugh Walker. 2005. “An Assessment of the Consumer Ethnocentric Scale (CETSCALE) in an Advanced and Transitional Country: The Case of Canada and Russia.” International Journal of Management 22 (4): 556-571.

  11. Latent variable approach Consumer ethnocentrism (National) products, first, last, foremost Real (national) buys domestic should always buy domestic Purchasing foreign is un-patriotic Buy domestic Keep country working Foreign products only out of necessity Curbs should be put on all imports Foreigners should not be allowed to sell Foreign should be heavily taxed In latent variable model, oval represents latent construct, square represents manifest indicator.

  12. Finding for Russia (National) products, first, last, foremost Real (national) buys domestic should always buy domestic Purchasing foreign is un-patriotic Buy domestic Keep country working cultural Consumer ethnocentrism Economic Foreign products only out of necessity Curbs should be put on all imports Foreigners should not be allowed to sell Foreign should be heavily taxed

  13. Example: National Identity • Defined as beliefs about importance of potential determinants of membership of the nation. • Theory suggests 2 dimensions • Civic: residence, citizenship, respect for law/institutions • Ethnic/ascriptive: birthplace, descent, religion • Common example in methodological literature Davidov, Eldad. 2009. “Measurement Equivalence of Nationalism and Constructive Patriotism in the ISSP: 34 Countries in a Comparative Perspective.” Political Analysis 17 (1) (December 21): 64–82. doi:10.1093/pan/mpn014. Heath, Anthony, Jean Martin, and TheesSpreckelsen. 2009. “Cross-national Comparability of Survey Attitude Measures.” International Journal of Public Opinion Research 21 (3) (September 21): 293–315. doi:10.1093/ijpor/edp034. Loner, Enzo, and PierangeloPeri. 2009. “Ethnic Identification in the Former Soviet Union: Hypotheses and Analyses.” Europe-Asia Studies 61 (8): 1341–1370. doi:10.1080/09668130903134798. Kunovich, Robert M. 2009. “The Sources and Consequences of National Identification.” American Sociological Review 74 (4) (August 1): 573–593. doi:10.1177/000312240907400404. Sarrasin, Oriane, Eva G. T. Green, André Berchtold, and EldadDavidov. 2012. “Measurement Equivalence Across Subnational Groups: An Analysis of the Conception of Nationhood in Switzerland.” International Journal of Public Opinion Research (October 18). doi:10.1093/ijpor/eds033.

  14. Measurement model for pooled ISSP data Civic Citizen-ship Respect laws Speak language Feel nat’l identity Born in nation Life-long resident Nat’l ancestry Nat’l religion Ethnic Source: Heath 2009

  15. Citizenship as ethnic Countries where citizenship rules restrictive, ascriptive Countries that are ethnically homogenous Includes most former Soviet countries, Austria, Switzerland Source: Heath 2009

  16. Citizenship as civic Countries where citizenship rules less restrictive Countries with historical ethnic diversity Includes Czech Republic, Spain, Australia, Israel Source: Heath 2009

  17. Formal tests for measurement invariance • Configural invariance (weak): factor structure equivalent (same items load on same latent variables) • Metric invariance (strong): factor loadings equivalent: necessary to compare relationships between constructs (e.g. regression coefficients). • Scale invariance (strict): factor loadings and intercepts equivalent; necessary to compare means • Partial invariance: at least two items load equally on each construct. Byrne, Barbara M., and Fons J. R. van de Vijver. 2010. “Testing for Measurement and Structural Equivalence in Large-Scale Cross-Cultural Studies: Addressing the Issue of Nonequivalence.” International Journal of Testing 10 (2): 107–132. doi:10.1080/15305051003637306. Cheung, Gordon W. 2008. “Testing Equivalence in the Structure, Means, and Variances of Higher-Order Constructs With Structural Equation Modeling.” Organizational Research Methods 11 (3) (July 1): 593–613. doi:10.1177/1094428106298973. Steenkamp, Jan‐Benedict E. M., and Hans Baumgartner. 1998. “Assessing Measurement Invariance in Cross‐National Consumer Research.” Journal of Consumer Research 25 (1) (June 1): 78–107. doi:10.1086/209528.

  18. What to do when invariance not achieved • Delete problematic groups or countries from comparison set • Delete problematic items from measurement model • Settle for partial invariance: configural invariance, plus at least 2 indicators per latent variable with equal loadings and/or intercepts

  19. Problem with formal tests of invariance • Formal strictness can undermine substance • “Just as cross-national researchers recognize that indicators may need non-literal translation to maximize the comparability between countries, constructs also may need non-literal translation.” (Medina et al 2009)

  20. Achieving functional equivalence • Statistical approaches • “Locally-conditioned models”: control for reasons for invariance (Medina 2009) • Introduce contextual predictor variables – i.e. directly model cross-group differences (Davidov 2012) • Qualitative context • Investigate sources of invariance through cognitive interviews, observation(Carnaghan 2011) • Davidov, Eldad, et al. 2012. “Using a Multilevel Structural Equation Modeling Approach to Explain Cross-Cultural Measurement Noninvariance.” Journal of Cross-Cultural Psychology 43 (4): 558–575. doi:10.1177/0022022112438397. • Carnaghan, Ellen. 2011. “The Difficulty of Measuring Support for Democracy in a Changing Society: Evidence from Russia.” Democratization 18 (3): 682–706. doi:10.1080/13510347.2011.563113.

  21. RESPONSE STYLE BIAS

  22. Response styles • Tendencies in responses, independent of true belief. • Types of response styles • Tendency to agree (acquiescence bias) • Tendency to moderate responses • Tendency to extreme responses • Create bias in likert scales (e.g. agree-disagree), yes/no questions • Problematic when tendency to acquiesce varies across groups to be compared Harzing, Anne-Wil. 2006. “Response Styles in Cross-national Survey Research A 26-country Study.” International Journal of Cross Cultural Management 6 (2) (August 1): 243–266. doi:10.1177/1470595806066332. Kieruj, Natalia D., and Guy Moors. 2013. “Response Style Behavior: Question Format Dependent or Personal Style?” Quality & Quantity 47 (1) (January 1): 193–211. doi:10.1007/s11135-011-9511-4. Tobi, Hilde, and Jarl K. Kampen. 2013. “Survey Error in an International Context: An Empirical Assessment of Cross-cultural Differences Regarding Scale Effects.” Quality & Quantity 47 (1) (January 1): 553–559. doi:10.1007/s11135-011-9476-3.

  23. Example: acquiescence in Kazakhstan • Survey experiments on tendency to agree among Kazakh vs. Russian respondents in Kazakhstan Javeline, Debra. 1999. “Response Effects in Polite Cultures: A Test of Acquiescence in Kazakhstan.” The Public Opinion Quarterly 63 (1) (April 1): 1–28. doi:10.2307/2991267.

  24. Possible causes of acquiescence bias • Cultural significance of hospitality, deference, avoiding offense • Uncertainty about answer; assume statements contain cues about correct answer • Cognitive burden • Must infer counterarguments • If agree with inferred counterargument, must disagree with statement (yes/no) • Satisficing due to fatigue, disinterest

  25. Example: freedom versus order • Version A: “People should be free to say whatever they want, even if what they say increases tensions in society.” • Version B: “Public order should be maintained above all, even if it requires limiting freedom of speech.” • Version C: “Certain people think that it is better to live in a society with strict order, even if it requires limiting freedom of speech. Others think that people should be free to say whatever they want, even if what they say increases tensions in society. Which view is closer to your own? Do you feel this way strongly or only somewhat?” • Response options for versions A & B are: strongly agree/ somewhat agree/ somewhat disagree/ strongly disagree • Version C is a “forced choice” format question

  26. Evidence for acquiescence bias In absence of bias, proportions should be the same for all 3 versions. Acquiescence bias overstates support for freedom in version A, understates support for freedom in version B.

  27. Comparison of Kazakhs & Russians VERSION A VERSION B VERSION C CONCLUSION: Acquiescence bias in Version A understates differences Between groups

  28. What to do • Avoid likert scales (esp agree/disagree) • Use forced choice formats • If using likert scale: Randomize direction of question

More Related