Problems in the Comparability and Harmonization of Measures of Community Size in ISSP Surveys

1. Problems in the Comparability and Harmonization of Measures of Community Size in ISSP Surveys Michael L. Smith, Ph.D. Institute of Sociology, Academy of Sciences of the Czech Republic

2. Overview • URBRURAL and NAT_SIZE: what are they supposed to measure and why are they important? • Major problems in both questions • Cross-national analysis of association between the variables: what can we learn? • Controlling for population composition, what effect do they variables have on key dependent variables? • Re-designing both questions: considerations for input and output harmonization

3. URBRURAL: Type of community (self assessment) E.g. (CZ): How would you describe the location you live in? 1 Urban, a big city2 Suburb, outskirt of a big city3 Town or small city4 Country village5 Farm or home in the country NAT_SIZE: Size of community E.g. (CZ): What size category does the community you live in belong to? [response categories not determined by ISSP] 1 1.2 Mill. (Prague)2 100.000 inhabitants and more, city3 50.000 - 99.999 inhabitants4 10.000 - 49.999 inhabitants5 5.000 - 9.999 inhabitants6 2.000 - 4.999 inhabitants7 1.000 - 1.999 inhabitants8 Less than 1.000 inhabitants Variables in question

4. Theoretical importance of variables • Variables reflect the importance of urban-rural contextual differences in shaping individual attitudes and behaviors • Lipset & Rokkan: Urban-rural cleavages emerged from the process of urban industrialization major impact on political behavior & party systems • Impact of urban-rural cleavages continue to be a major research topic • Urban-rural cleavage is arguably declining in relevance in West, but still very important in developing nations (i.e. new ISSP members)

5. Key differences in variables

6. Different approaches to NAT_SIZE • Some countries did not ask NAT_SIZE: Chile & Venezuela • Germany, Austria, USA and Philippines did not have question, but used a coder to enter objective pop. size • Great Britain used administrative variable on population density (1 = Less than 2.8028 persons per hectare, etc) • Switzerland used postal codes of respondents & official data to construct CH_SIZE • Dominican Republic and Turkey have coder code for “province” of interview • Australia used one question for both AU_SIZE and URBRURAL • South Africa asked about “environmental milieu”: 1 = Urban Settlement, 2 = Tribal Settlement, 3 = Small Holding, 4 = Informal Settlement, 5 = Hostel

7. Some countries also constructed URBRURAL from NAT_SIZE

8. Other countries also did that (e.g. Russia) even if it was not reported in ISSP documentation

9. Debate in ISSP • ISSP Demographics & Methods Group: • In 2008, DMG raised concerns that the questions are too subjective, and do not adequately reflect the geographic / territorial context of respondents • DMG raised concerns over the predictive value of the variables • DMG suggested dropping both variables • ISSP General Meeting, Vienna 2008: • Members voted 26 to 3 to keep URBRURAL • Members voted 18 to 11 to drop NAT_SIZE, and is no longer a required demographics variable in ISSP

10. Assessing URBRURAL and NAT_SIZE from the bottom up • Data and methods: • Followed approach of Prof. Christof Wolf (GESIS) in his excellent analysis of the variables for Germany (prior to ISSP 2009 meeting)  replication for other countries • First, analysis of variance between URBRURAL and NAT_SIZE for each country: good measurement should lead to consistent associations between variables across years • Second, multiple classification analyses of URBRURAL and NAT_SIZE on a range of dependent variables in ISSP 2008 and ISSP 2007 in different countries (also controlling for education, sex and age), as a way to assess the significance of the variables across countries

11. Association of URBRURAL and NAT_SIZE for countries that use self-assessment for the latter

12. Association of URBRURAL and NAT_SIZE for countries that treat the latter as an administrative variable

13. Key lesson from the analysis • On the whole, ISSP members that treat NAT_SIZE as an administrative variable have more consistent data in terms of the association between NAT_SIZE and URBRURAL over time. • ISSP members using self-assessment for NAT_SIZE tend to have higher degrees of association between the variables, probably due to bias of question order and suggestively phrased response categories. CZ case: • 1 = 1.200.000 inhabitants (Prague) • 2 = City with 100.000 inhabitants and more • 3 = 50.000-99.999 inhabitants, large city • 4 = 10.000-49.999 inhabitants, larger town • 5 = 5.000-9.999 inhabitants, small town • 6 = 2.000-4.999 inhabitants, larger village • 7 = 1.000-1.999 inhabitants, village • 8 = Village with less than 1.000 inhabitants

14. ISSP 2008: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

15. ISSP 2008: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

16. ISSP 2008: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

17. ISSP 2008: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

18. ISSP 2007: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

19. ISSP 2007: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

20. ISSP 2007: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

21. ISSP 2007: Results of multiple classification analyses: gross and net effects of URBRURAL and NAT_SIZE

22. Key lessons from the analysis • Across a range of dependent variables, the urban-rural cleavage tends to be strongest in developing countries (Mexico, South Africa, Taiwan) and weakest in some of the Western countries • In most cases examined, control variables reduce the effect of URBRURAL and NAT_SIZE on the margin, but do not eliminate their importance as explanatory variables • Overall, the analysis has shown that these two variables continue to show strong effects in ISSP countries, and should remain in the ISSP demographics module.

23. Considerations for harmonization: URBRURAL • There are many opportunities for cross-national analyses of urban-rural cleavages. But for those analyses to work, ISSP members must implement the full set of response categories (i.e. not reduce them) • The question MUST be based on respondent self-assessment in all countries, without undue interference of the interviewer (the question should be “subjective” in all countries) • The practice of computing responses based on NAT_SIZE (and vice-versa) must stop.

24. Considerations for harmonization:NAT_SIZE • For the variable to be reintroduced into ISSP, it should be decided whether the question should be a demographic (respondent assessment) or an administrative variable • The analysis above provides stronger support to the administrative variable option: • It is easier to monitor how the data is measured • It removes the “subjective” dimension of the question, making it a pure objective demographic characteristic • It avoids potential problems 1) in bias due to question order and 2) suggestive questions.

25. Considerations for harmonization:NAT_SIZE • Some ISSP members will complain of the difficulty of collecting the data  but in most cases, the fielding agency should be able to identify community size during the sampling frame • To avoid problems in the identification of specific municipalities, response categories should still be used • Those response categories should be broad so that they are appropriate for both countries with many small villages (France, Czech Rep.) but also for those with highly urban populations (Taiwan, Korea).

26. Considerations for harmonization:NAT_SIZE • Lastly, we need to have a debate about what should be measured by NAT_SIZE: • The population of the territory of the local government (municipality, commune, etc)? Or, • The population of the smallest distinct ‘settlement’ of the respondent, regardless of whether or not it belongs to a larger political-administrative unit • I personally believe the second option is far superior, as it avoids problems when municipalities encompass very large territorial areas (as in Poland, Italy, etc). However, a feasibility analysis needs to conducted about whether data is attainable on sub-municipal units in ISSP countries.

27. Thank youMichael.Smith@soc.cas.cz