Nonresponse adjustment in the European Social Survey (ESS)

1. Nonresponse adjustment in the European Social Survey (ESS) Ineke Stoop (SCP-Netherlands) Jaak Billiet (Uni Leuven-Belgium) Achim Koch (Gesis-Germany)

2. European Social Survey Content Methodology To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond Input harmonisation • To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions

3. Basicfacts • Biennial survey • 30+ countries • 2.000 interviews/country • Fieldwork organisation/ country • Probability sampling • Design weights • No substitution • Face-to-face • Fieldwork monitoring • Guidelines interviewer training • Target noncontact rate: 3% • Minimum number of calls • Timing of calls (evening, weekend) • Target response rate: 70% • Guidelines response enhancement • Incentives/Brochure • Refusal conversion • Contact form • Interviewer observations • Call records • Standard response rate calculation

4. Factors causingvariancein nonresponse bias • Survey climate • Populationcharacteristics • Difficulty to obtain response • House effects • Fieldwork staff • Fieldwork strategy • Efforts to obtain high response rates • Sampling frames • Realised response rates • Nonresponse composition • noncontact/refusal/notable (language) • Reasonsforrefusal • Presence of auxiliary data to adjustfor nonresponse bias

5. Response rates ESS

6. Assessing and correcting for bias in a cross-national study • Ideal: samewayeverywhere • Practice: availability, accessibilityauxiliary data differs • Sampling frames differs • Usuallynoaccess to population register • Qualitypopulationstatistics • Quality paradata • Ease of response (wave data) • Onlyapproachesthat COULD beused in every country?

9. Bias: top and bottom of 45 items

10. Poststratification in ESS

12. Definition reluctant respondents/converted refusals • Initialcooperationrate? • Refusalconversionstrategy • None • Easy • All • Largedifferences in strategies and resultsacrosscountries • Ifthere are few: high initial, good job orpoor start?

13. Converted refusals in ESS 2

14. Evaluationrefusalconversionapproach ESS Positive Negative Privacy regulations Quality call records Different initial response rates Different efforts across countries Different success rate refusal conversion Aimed at different groups (all/easy) Few countries only • Enhances response rates • Conveys message that every respondent is important

16. Type and quality of data • Characteristics neighbourhood and dwelling • Good data in about half of the ESS countries • Better in later rounds • Estimation of age and sex • Good data in only a few of the countries • Is refuser target person? • Conflicting instructions contact form

17. Content • Variables measured • Type of dwelling (apartment/detached/...) • Physical state of building/dwellings in neighbourhood • Litter or rubbish in immediate area • Vandalism, graffiti or damage to property • Poorer condition correlates with lower education

18. Some results • Living in apartment increases likelihood to refuse and not to be contacted (exception AT, why???) • More likely to refuse and noncontact when physical condition is worse (exception AT, why???) • Litter and vandalism in environment: weak effects (exception AT) • Interaction between dwelling type (apartment) and physical condition in ES and SK (more likely not to participate and noncontact if apartment in bad state)

19. Evaluation interviewer observation approach ESS Positive Negative Missing data Poor quality of coding Neighbourhood scores comparable across countries? Neighbourhood data proxy for household/ individual data • All sample units used (cooperative, refusals, non contacted) • Moderate correlation moderately with target variables as social status (education)

21. Data and methodNRS sample design in 4 countries ESS 3

22. Basic questionnaire Very short: 7 questions Short: 9 additional questions Sex Year of birth TV watching Voluntary work Social trust Satisfied with democracy Trust in politics Immigration good/bad for country (Reasons for refusal) • Work situation • Highest level of education • # of members in household • Frequency of social activities • Feeling (un)safe • Interest in politics • Attitude towards surveys

23. Many different comparisons to be made Doorstep Follow-up • Cooperative main • Reluctant main • Cooperative doorstep (calibrated)

24. DQS Belgium: different cooperative?

25. Results Belgium doorstep approach Criterium 1 Criterium 2 (coop/refusal) Weighted cooperative respondents do not differ from unweighted cooperative respondents

26. Do survey responses differ between nonrespondent and cooperative respondents in follow-up and main survey in Norway?

27. Do survey responses differ between nonrespondents and cooperative respondents in follow-up and main survey in Norway?

28. Are the distributions of key variables independent by the types of respondents (nonrespondents and cooperative main) in Norway? (criteria 2)

29. Are the distributions of key variables independent by the types of respondents (nonrespondents and cooperative main) in Norway? (criteria 2)

30. Is it possible to correct nonresponse bias on the basis of nonresponse survey (NRS)? • On the basis of keyquestionsonsocio-demographic, behavioral & attitudinal variables, non-response bias canbemeasured • whichinformation to take is crucial • Response propensitiescantake different form; • Basedonthisapproach, nonresponse bias canbecorrectedfor (butnotfor all variables)

31. Evaluation doorstep core question approach ESS Positive Negative Context effect Refusals only Remaining refusals doorstep questionnaire Few questions only • Can correct for refusal • Relatively inexpensive • Combine with paradata/neighbourhood data

32. Evaluation follow-up core question approach ESS Positive Negative What groups to compare (initially reluctant?) High nonresponse in follow-up study Different initial response rates Few questions only • Can correct for nonresponse • Include all nonrespondents • Combine with paradata/neighbourhood data

33. What to do in 30+ country study? • Initial situation differs • Possibilities differ • Use national best auxiliary information • Study bias in many different ways • Optimal national adjustments threaten comparability? • Provide nonresponse weights to users?

34. More onthis topic