Mixed-mode data collection Recent developments at Statistics Netherlands

1. Mixed-mode data collectionRecent developments at Statistics Netherlands Gabriël van Dam

2. Outline • Background and motivation • Quality and costs • Redesigns • Mode effects • Conclusions and discussion Seminar on Statistical Data Collection – September 26, 2013

3. Background Quality & costs Redesigns Mode effects Discussion Developments at Stat Netherlands Generally, focus on costs: Project Redesign of Social Surveys. 2007 – 2012 Focus on quality: Project Mode Effects in Social Surveys, 2010 – 2012 Adaptive survey designs: Studies for CVS and LFS, 2011 - 2013 Seminar on Statistical Data Collection – September 26, 2013

4. Background Quality & costs Redesigns Mode effects Discussion Focus on costs At Statistics Netherlands, five main questions: • What are response rates for web and Mixed-mode designs? • Can we do Mixed-mode designs with web? • What should be lengths of questionnaires and how to transform complex survey items? • Are new statistics comparable and stable? • What about undercoverage of web? Seminar on Statistical Data Collection – September 26, 2013

5. Background Quality & costs Redesigns Mode effects Discussion Redesign of Social Surveys Conclusions to main redesign questions Response rates: • Web cannot be single mode • MM designs have response rates and representativeness similar to CAPI • Web recruitment rates lower than CATI/CAPI Implementation: • Major changes to administration systems • Complex monitoring of sample cases • Data collection period was lengthened Seminar on Statistical Data Collection – September 26, 2013

6. Background Quality & costs Redesigns Mode effects Discussion Redesign of Social Surveys Questionnaires: • Questionnaires had to be bisected • Complex items simplified or replaced by registries Comparability and stability: • Parallel runs conducted for all redesigns • Quality reports before and after redesigns in order to support explanation of method effects • Web shows unstable response rates • Large mode effects are possible Coverage web: • Web access rate at home is 90% • Undercoverage based on age, income, ethnicity and urbanization degree Seminar on Statistical Data Collection – September 26, 2013

7. Background Quality & costs Redesigns Mode effects Discussion Context – LFS and CVS redesigns LFS LFS LFS <2010 2010 – 2012 >2012 web CAPI CATI CAPI CATI CAPI CVS (SN and public) CVS <2011 >2012 web+ web+ web + paper CATI CAPI CATI Seminar on Statistical Data Collection – September 26, 2013

8. Background Quality & costs Redesigns Mode effects Discussion Mode effect components Data collection steps: Persons need to be reached (coverage) Persons need to respond (response) Persons need to provide valid answers (measurement) Total relative mode effect (telephone – face-to-face): Mtel(y) = ytel,tel,tel – ytot,f2f,f2f coverage response measurement Seminar on Statistical Data Collection – September 26, 2013

9. Background Quality & costs Redesigns Mode effects Discussion Mode effects - conclusions CVS: • Telephone most optimistic and web most pessimistic • Measurement effect is dominant effect • Mode effect very strong on victimization topics • Questionnaire needs redesign LFS: • Mode effect is mix of effects • Selection effect can be explained using register variables General: • Mode-specific measurement effects will lead to method effects in redesigns Seminar on Statistical Data Collection – September 26, 2013

10. Background Quality & costs Redesigns Mode effects Discussion Mode effects - implications CVS: • Weighting only removes a small part of the mode effect • Mode effects must be avoided by design and/or must be stabilized by calibration to fixed mode totals LFS: • Weighting removes a large part of the mode effect • Remaining mode effect is difference in measurement Seminar on Statistical Data Collection – September 26, 2013

11. Background Quality & costs Redesigns Mode effects Discussion Conclusions • Web cannot be used as a single mode for social surveys and lead to MM designs • MM surveys including web are feasible after some learning period but require investments in infrastructure, monitoring and pilot studies • Inclusion of web as a survey mode implies a redesign of survey questionnaires • Mode effects between web and interviewer modes can be large Seminar on Statistical Data Collection – September 26, 2013

12. Background Quality & costs Redesigns Mode effects Discussion Recommendations Reduce, avoid and adjust: • Reduce measurement effects by careful questionnaire design • Avoid contrasting modes and assign resources to subpopulations that show strongest mode effects • Adjust for method effects by stabilizing or calibrating the distribution of respondents over modes Seminar on Statistical Data Collection – September 26, 2013

13. Background Quality & costs Redesigns Mode effects Discussion Discussion • Web response rates are still much lower than other modes. Increase of rates should be possible? • How to deal with household data collection in web surveys (in terms of self and proxy reporting)? • How to increase web recruitment rates for subsequent waves? • How to stabilize statistics in MM surveys including web? Questions? Email: gdam@cbs.nl Seminar on Statistical Data Collection – September 26, 2013