Mixed-mode survey designs Recent developments at Statistics Netherlands

1. Mixed-mode survey designsRecent developments at Statistics Netherlands Barry Schouten - Statistics Netherlands and University of Utrecht The 4th International Workshop on Internet Survey Methods September 12 – 13, Daejeon

2. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Context – LFS and CVS redesigns LFS LFS LFS <2010 2010 – 2012 >2012 web CAPI CATI CAPI CATI CAPI CVS (SN and public) CVS <2011 >2011 web+ web+ web + paper CATI CAPI CATI 4th International Workshop on Internet Survey Methods, 12-13 Sept

3. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Quality and costs web paper CATI CAPI costs paperCAPI CATI web coverage rate CATI CAPI paper web response rate CATI paper web CAPI representativeness ? measurement 4th International Workshop on Internet Survey Methods, 12-13 Sept

4. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Quality and costs • Mixed-mode survey designs including web • Focus on costs: Decrease budgets while preserving • comparability and to lesser extent accuracy (NSI`s) • Focus on quality: Increase accuracy by mitigating • weaknesses of modes by strengths of other modes • (academic) • Trade-off between quality and costs can be made mathematically more rigourous by applying adaptive and responsive survey designs 4th International Workshop on Internet Survey Methods, 12-13 Sept

5. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Focus on costs • At Statistics Netherlands, five main questions: • What are (subgroup) response rates for web and MM designs? • Can we do MM designs with web? • What should be lengths of questionnaires and how to transform complex survey items? • Are new statistics comparable and stable? • What about the undercoverage of web? 4th International Workshop on Internet Survey Methods, 12-13 Sept

6. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Focus on quality • Two prerequisites to combining modes: • There is a preferred benchmark design or true value • Mode effects can be disentangled into mode-specific coverage, mode-specific response and mode-specific measurement effects 4th International Workshop on Internet Survey Methods, 12-13 Sept

7. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Adaptive survey designs Steps: • Choose quality and cost functions • Identify candidate strategies • Identify population subgroups (registries, paradata) • Estimate cost and quality parameters • Optimize allocation of subgroups to strategies: • either, maximize quality given constraints on cost • or, minimize cost given constraints on quality • Monitor data collection closely, and possibly repeat steps Here: strategies are survey modes 4th International Workshop on Internet Survey Methods, 12-13 Sept

8. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Developments at Stat Netherlands Generally, focus on costs: Project Redesign of Social Surveys, 2007 – 2012 Focus on quality: Project Mode Effects in Social Surveys, 2010 – 2012 Adaptive survey designs: Studies for Crime Victimisation Survey and Labour Force Survey, 2011 - 2013 4th International Workshop on Internet Survey Methods, 12-13 Sept

9. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Redesign of social surveys • Conclusions to main redesign questions • Response rates: • Web cannot be single mode • MM designs are needed • MM designs have response rates similar to CAPI • Web recruitment rates lower than CATI/CAPI • Implementation: • Major changes to administration systems • Complex monitoring of sample cases • Data collection period was lengthened • Questionnaires: • Questionnaires had to be bisected • Complex items simplified or replaced by registries 4th International Workshop on Internet Survey Methods, 12-13 Sept

10. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Redesign of social surveys • Comparability and stability: • Parallel runs conducted for all redesigns • Quality reports before and after redesigns in order to support explanation of method effects • Web shows unstable response rates and shares of modes to response may vary • Large mode effects are possible • Ad hoc method: calibrate to fixed mode totals • Coverage web: • Web access rate at home is 90% • Undercoverage based on age, income, ethnicity and urbanization degree • Forms input to adaptive survey designs 4th International Workshop on Internet Survey Methods, 12-13 Sept

11. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Sample = 8800 persons Wave 1 CAPI CATI web paper Response Nonresponse Wave 2 CAPI (+CATI) Mode effects in social surveys Starting point: Selection effects and measurement effects are estimated relative to face-to-face interviews 4th International Workshop on Internet Survey Methods, 12-13 Sept

12. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Mode effects LFS 4th International Workshop on Internet Survey Methods, 12-13 Sept

13. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Mode effects - conclusions CVS: • CATI most optimistic and web most pessimistic mode • Measurement effect is dominant effect • Mode effect very strong on victimisation topics • Questionnaire needs redesign LFS: • Mode effect is mix of effects • Selection effect can be explained using register var`s General: • Mode-specific measurement effects will lead to method effects in redesigns 4th International Workshop on Internet Survey Methods, 12-13 Sept

14. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Mode effects - implications CVS: • Weighting only removes a small part of the mode effect • Mode effects must be avoided by design and/or must be stabilized by calibration to fixed mode totals LFS: • Weighting removes a large part of the mode effect • Remaining mode effect is difference in measurement • Additional evaluations of LFS parallel runs required to identify subgroups with large measurement effects 4th International Workshop on Internet Survey Methods, 12-13 Sept

15. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Adaptive survey design – CVS Starting point • CAPI is benchmark for CO + NR • Both CAPI and paper evaluated as benchmark for ME • Budget is current CVS • Additional samples have limited budget Objective: Minimize measurement effect wrt benchmark Input parameters: Decomposition of mode effect for gender, household type, age, ethnicity, urbanization Strategies: web, paper, CATI, CAPI, web→CATI, web→CAPI, paper→CATI, paper→CAPI 4th International Workshop on Internet Survey Methods, 12-13 Sept

16. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Conclusions adaptive design CVS Findings evaluation CVS: • Combination of CATI and non-interviewermodes leads to unstable estimates • Subpopulations show variation in measurement effects • Measurement effects are for all subpopulations larger and in most cases much larger than the selection effect • Largest measurement effects: the number of crimes for single person households, women and urbanized areas Optimal designs given benchmark mode: • Paper: a combination of paper and web • CAPI: an adaptive survey design including all modes Given limited budget additional samples web + paper selected 4th International Workshop on Internet Survey Methods, 12-13 Sept

17. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Conclusions • Web response rates too low and lead to MM designs • MM surveys including web are feasible after some learning period but require investments in infrastructure and monitoring • MM surveys including web are considerably cheaper • Mode effects between web and interviewer modes are sometimes large • Comparability of statistics enforced by design of data collection and adjustment for nonresponse 4th International Workshop on Internet Survey Methods, 12-13 Sept

18. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Discussion • Web response rates are still much lower than those for mail. Increase of rates should be possible? • How to deal with household data collection in web surveys (in terms of self and proxy reporting)? • How to increase web recruitment rates for subsequent waves? • How to stabilize statistics in MM surveys including web? 4th International Workshop on Internet Survey Methods, 12-13 Sept

19. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Mode effect decomposition - assumptions • The response to wave 2 is similar to the responseto a regular CAPI survey • The answering behaviour to wave 2is CAPI • The nonresponse to wave 1 relative to wave 2, is missing at random • The mode effects in wave 2 between CATI and CAPI have a negligible impact 4th International Workshop on Internet Survey Methods, 12-13 Sept

20. Redesigns • Quality - costs • Developments • Redesign • Mode effects • ASD • Discussion Mode effects CVS victim #crimes unsafe nuisance 4th International Workshop on Internet Survey Methods, 12-13 Sept

21. Research Q`s • Design • Results • Mode effects • Weighting • Mode strategy • Discussion Mode effects in mixed-mode designs 4th International Workshop on Internet Survey Methods, 12-13 Sept