From face-to-face to web in a cross-national study

1. From face-to-face to web in a cross-nationalstudy Ineke Stoop SCP/CST

2. ESS • Academic survey in 30+ European countries • Attitudes, values and beliefs • Biennial, start in 2002 • 1+ hour, face-to-face, effective sample size ~ 1500 • Funders • National Science organisations • European Commission, Framework Programmes • European Science Foundation • Management model • Central coordination • National implementation

3. Aims ESS To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions Provide data on beliefs, attitudes and values for scientific and policy making purposes Measure attitude change in a changing Europe To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond Collect data according to highest standards Generate methodological research Develop and disseminate new best practices Develop and improve social survey research infrastructure in Europe

4. Content questionnaires ESS Everyround Roundspecific R1 Immigration Citizenship, involvement and democracy R2 Family, work and well-being Opinions on health and care- seeking Economic Morality R3 Personal & Social Well-being The Timing of Life R4 Ageism Welfare attitudes R5 Family, work and well-being in times of economic recession Trust in police and courts R6 Personal & Social Well-being Understanding, evaluation democracy • Socio-demo-graphic items • household composition • education • housing • occupation • social structure • Attitudes and behaviour patterns • religious affiliation and identity • ethnic and national identity • political and social trust • party affiliation and voting behaviour • media consumption • value orientations • social exclusion

5. Access ESS Round 1-5 available Fully documented Free access Tables and graphics online, or download data Training package for young researchers Everything available All deviations documented

6. ESS top data user countries (>50) (June 2012) (N=48,973)

8. Methods: high quality • Strictly random sampling • No substitution, no proxy • Population: all inhabitantsaged 15+ • High target response rates • 70% target response rate, max. 3% noncontact • Face-to-face fieldwork • Gold standard, high response rates, universally applicable • Increasingly expensive • Rare in some countries • Pilots, pretesting and fieldwork monitoring • Analysis of cross-national differences • Nonresponse bias, measurement error, sampling design effects

9. Methods: central coordination Central specifications for fieldwork Close monitoring national fieldwork (in advance and during fieldwork) Sampling design Translation procedures Guidelines response enhancement Guidelines interviewer training Contact forms, response calculation Context, events

10. Can we comparecountries? Attitudes, values, beliefs Survey results Survey design, methodology, implementation

11. Harmonisation strategies (Körner and Meyer, 2005) Harmonised concept Harmonised concept Harmonised concept Measurement procedure Measurement procedure Measurement procedure National survey/ Specific concept National survey/ Specific concept National survey/ Specific concept Input harmonisation Ex-ante output harmonisation Ex-post output harmonisation

12. Output harmonisation • Ex post output harmonisation • Use existing sources • Ex ante output harmonisation • Set up new survey (or develop new instrument) • Output harmonisation • Best national quality • Or, national survey tradition • But, what about optimal comparability

13. Output harmonisation: problems • Interview mode • Context question • Lay-out questionnaire • Question text • Introductory text • Answer scales • Handling don’t know • Size of boxes • Use of illustrations

14. Input harmonisation • Input harmonisation • Design new survey • Optimal comparability • Or, are identical methods and instruments really equivalent in different countries? • But, what about optimal quality

15. From face-to-face to web in the ESS • Mixed mode research and experiments • Different survey modes over time • Different survey modes across countries • Mixed mode within countries: different composition over time • Which modes: • Face-to-face • Telephone • Web

16. ESS studies • Martin, Peter (2011) A Good Mix? Mixed Mode Data Collection and Cross-national Surveys. ASK Research & Methods, Vol. 20 (1, 2011) 5–26. • Martin, Peter (2011) What makes a good mix? Chances and Challenges of Mixed Mode Data Collection in the European Social Survey. CCSS Working Paper No. 02. • Martin, Peter, and Lynn, Peter (2011) The Effects of Mixed Mode Survey Designs on Simple and Complex Analyses. CCSS Working Paper No. 02. • Roberts, Caroline (2007) Mixing Modes of Data Collection in Surveys: A Methodological Review. ESRC National Centre for Research Methods. NCRM Methods Review Papers, NCRM/008.

17. From face-to-face to web in the ESS • Coverage • Sampling • Nonresponse • Questionnaire design • Role interviewers • Measurement • Analysis • Cross-national comparison

18. Coverage

19. Internet access and broadband connections in households, EU27 (%)

20. Individuals who used the internet at least once a week, 2011 (%)

21. Individuals who used the internet on average at least once a week, by sex, age and education, EU27, 2011 (%)

22. Sampling principles • Identical population definition • Complete coverage • No upper age limits • (Institutional population) • (Non-native language speakers) • Probability sampling • Effective sample size • Optimal sampling design in each country • Address, household, individual • Address and household: selection of target respondent in the field

23. ESS R2 sampling frames (Häder and Lynn) Simple random sampling, or simple stratified sampling (no design effect)

24. Comparison sampling frames • Achim Koch (GESIS) • In household and address samples: selection of respondent by interviewer • Leads to overrepresentation of women in household/address samples • Related to higher response rates • Close control on who answers the questionnaire in a household even more problematic in web survey

25. ESS 1 – 4:Sample ofindividuals: Bias X response rate

26. ESS 1 – 4:Sample ofhhs/addresses: Bias X response rate r = .44

27. Nonresponse ESS • Response targets • 70% response rate • 3% noncontact rate • Standards in • Response rate calculation • Contact form • Nonresponse analysis

28. Response efforts and monitoring • Guidelines response enhancement • Incentives/Brochure • Refusal conversion • Call schedule requirements • Minimum number of calls • Timing of calls (evening, weekend) • Contact form • Interviewer observations • Call records

29. Response rates ESS (selection all rounds)

30. Response rates over time

31. Nonresponse • Response rate highest for face-to-face • Functional illiteracy (differs across countries) • Nonresponse analysis • What happens in web • Noncontact, refusal, not able • Reasons for refusal • Interviewer judgment • Refusal conversion

32. Questionnaire design • Questionnaire length • Split questionnaire • Matrix design • Over time • Question design • Unimode • Developed for face-to-face

33. Role interviewers • Select respondents (+) • Obtain cooperation (+) • Conduct interview • Explain questions (-) • Explain process (+) • Sensitive questions (+/-) • Validity answers (+) • Social desirability (-)

35. Respondent understands questions

37. Measurement • Mode effects • Part of analysis • No simple adjustment • Mixed modes: • Selection effects and mode effects • Always comparison face-to-face • Mode in analyses

38. Cross-national comparison • Comparison over time • Mode change or time change • Comparison across countries • Mode difference or cultural difference • Mixed modes within countries • Selection effect or mode effect • Mode distribution over time • Feasibility of web for ESS (length, type of questions)

39. Thank you for your attention