Survey Design and Cross cultural Issues

1. Survey Design and Cross cultural Issues Avinish Chaturvedi Carlos Torelli

2. Agenda • Brief Review of the readings • Methods and Data Analysis for Cross-Cultural Research • Discussion

3. Heine et al., (2002) • Reference group effect: people from different cultural groups use different referents in their self-reported values (i.e., compare with different others) • Low Individualism -------------------- High Individualism  • ------------- Japanese High (7) ------ American High (7)  An apparent no difference is actually a significant difference

4. Wong et al., (2003) • Problems with Likert scales that contain a mixture of positive-worded items (PWI) and reverse-worded items (RWI) in cross-cultural research. • Why use RWI?: • Reduce acquiescence bias. • General problems with RWI? • May negatively impact internal consistency. • May disrupt dimensionality (RWI loading in a separate factor).

5. Brislin chapter: • Why do we need to change/modify the survey instrument in a different culture • Linking it with previous discussion • What factors change across cultures • Why not to use existing scales

6. Types of cross cultural studies: Van de Vijver chapter • Generalizability studies: The design of a generalizability study is usually a replication of the original study. - When the goals of cross cultural studies are defined as both delineating and explaining cross cultural differences, generalizability studies capitalize on first and often ignore the second.

7. Types of cross cultural studies: Van de Vijver chapter • Studies of psychological differences These are often based on a less elaborate theoretical framework than are generalizability studies

8. Types of cross cultural studies: Van de Vijver chapter • Theory driven studies - These studies test a theory of cross cultural differences. Theory driven studies are more systematic. Such studies are designed and carried out to critically test a hypothesis. These studies provide powerful tets of theories of cross cultural differences, which is one of the main goals of cross cultural research.

9. Types of cross cultural studies: Van de Vijver chapter • External validation studies - These studies take observed cross cultural level or structural differences as their starting point and scrutinize these differences either by exploring their antecedents or by testing interpretations of these differences.

10. Cross cultural issues • Researchers run the risk of missing aspects of a phenomenon as viewed by people in other cultures. • They risk imposing conclusions based on concepts which exist in their cultures but which are foreign, or at least partially incorrect, when used in other culture.

11. Emic and Etic distinction • How to determine what attributes of a construct are static, i.e., unchangeable across cultures • What a priori measures can be employed in this regard • Example from Brislin chapter: Authoritarian and conservatism. • What kind of questions are more sensitive to cultural variations?

12. Method and Data analysis • In cross cultural studies what kind of sampling measures should be taken • Convenience Sampling: Studies adopting this sampling scheme often fall into the category of psychological differences studies

13. Method and Data analysis • Systematic Sampling: Cultures are selected in a systematic, theory guided fashion. These studies usually fall into the categories of theory driven or generalizability studies. Cultures are selected in this procedure because they represent different values on a theoretical continuum • Random Sampling: It involves the sampling of a large number of cultures randomly. This strategy is preferable for generalizability studies, in which a universal structure or a pan cultural theory is evaluated.

14. Summary

15. Method and Data analysis • Choosing Cultures: How does choice of cultures in a research design changes when: a. we are looking for similarities b. we are looking for differences.

16. Method and Data analysis • Administration of instruments: tester/interviewer testee/interviewee interaction between two response procedures stimulus materials

17. Method and Data analysis • Remedies: A priori and post priori techniques. Similarly prior and post hoc remedies can be used to alleviate problems of sample incomparability. Application of a monotrait multimethod matrix in order to examine the influence of response procedures is useful What are the other ways through which this effect can be minimized??

18. Method and Data analysis • Changing the survey instrument Application Adaptation Assembly In which scenario, each of above three technique becomes useful?

19. Method and Data analysis • Validity Enhancement The major criterion in the choice of application, adaptation and assembly is the type of bias expected. If there are serious concerns that construct bias could play a role, adaptation or assembly should be chosen. Indigenization which aims at maximizing the appropriateness of psychological theories and instruments to local cultures, will often amount to the assembly of new instruments

20. Method and Data analysis • Obtaining linguistically equivalent instruments: Translation and back translation Committee approach

21. Method and Data analysis • Translation Translation is more than producing text in another language. Translators should know or be made aware of the linguistic and cultural differences that could influence responses to translated or adapted instruments.

22. Translation • Guidelines for translating and adapting psychological and educational instruments: • What are the other translation issues involved

23. Study 1 – Experts’ Ratings • Sample: Japanese specialists living in North America • What were the questions? • How much items in Singelis (1994) scale characterize Japanese or Americans • Findings: • Experts were in agreement with the common view for all 23 items (high face validity). • These findings suggest that the use of different reference groups can obscure cross-cultural differences (Americans use other Americans, and East Asians other East Asians).

24. Study 2 – Manipulation of Reference Groups • Sample: Canadian biculturals with knowledge about Japan and Canada. • Instruments: • Singelis (1994) in standard form. • Answers in comparison with most Japanese. • Answers in comparison with most Americans. • Findings: • Weak support for the common view using standard format. • When comparing with people from the other culture: strong support for the common view. • When comparing with people from their own culture: evidence from acculturation.

25. Study 3 – Within-culture • Sample: Asian Canadians vs. European Canadians, and returnee Japanese vs. Japan-bound Japanese • Common referent for each group. • Findings: • European Canadians scored higher in independence and lower on interdependence than Asian Canadians • Returnee Japanese scored higher on independence and lower on interdependence than Japan-bound Japanese

26. Discussion • Likert scales capture one’s feelings relative to a comparison group or shared norm? • Is it always the case? The case of self-esteem? • When can Likert scales assess one’s feelings without a referent? When do people use less social comparison for answering (i.e., introspection)? • Remmedial measures: • Measure at the cultural level • Prompt comparisons with an standard (average peer) • Responses to information that is consistent/inconsistent with implicit theories (new information) • Use of behavioral measures • Items with concrete, objective response options • Implications from the reference-group effect: • Use of subjective Likert scales most valid for identifying differences within rather than between groups  What do you think?

27. The MVS • The MVS in cross-cultural settings: • Good reliability and validity in the U. S. • Questionable reliability and validity abroad. • Why the problems outside the U. S.? • Negations and contradictions can be confusing • Languages can employ different ways of making negation or contradiction. • Differences in how PWI and RWI are interpreted (cultural norms regarding agreeableness?)

28. MVS across cultures – Study 1 • Countries: the U. S., Singapore, Japan, Korea, and Thailand. • Prediction: • Negative correlation with life satisfaction in wealthy countries only. • Positive correlation with individualism across all countries • Findings: • MVS specified as a 2nd. Order factor with 3 dimensions (happiness, centrality, and success) showed a poor fit  systematic error? • PWI and RWI correlated in the U.S., but weakly or uncorrelated elsewhere RWI being responded differently? • Divergent correlations for materialism and life satisfaction • Revised Model: include two method factors  fit comparable to those obtained in the U.S., but construct validity still an issue • Support for the notion that RWI might have led to the poor initial performance

29. MVS across cultures – Study 2 • Objectives: • Assess conceptual meaning of materialism. • Show that other scales that use PWI and RWI have the same problems. • Evaluate alternative scales  interrogative format • Predictions: • Show conceptual equivalence by using adjectives • Other scales that include RWI should show two-factors structures • Inquiring about a respondent’s position on an issue would minimize “agreement for the sake of being polite” • What do you think about the way conceptual equivalence was assessed? • Does it assess the 3-dimensions? • What does the evidence that other scales show two factors mean? • Does it prove that agreeableness is the underlying issue? What about the findings in the general discussion?

30. Discussion • Acquiescence vs. “substantive cultural differences”? • Are the results driven by acquiescence or not? • Are the meanings of the MVS items different across cultures due to religion and beliefs? If so, what is the explanation for the results with the other 4 scales? • Can there be any referent group effect? • Recommendations: • Use only PWI • Use interrogative formats • Which one is better?