1 / 10

Adaptive mixed-mode design WP1

ROME April 11 th | 12 th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP. Adaptive mixed-mode design WP1. Barry Schouten Statistics Netherlands (CBS). Adaptive mixed-mode survey design (ASD).

andrev
Download Presentation

Adaptive mixed-mode design WP1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ROME April 11th | 12th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP Adaptive mixed-mode design WP1 Barry Schouten Statistics Netherlands (CBS)

  2. Adaptive mixed-mode survey design (ASD) • Adaptive survey design optimizes quality-cost trade-offs by differentiating effort to different (relevant) population strata. • In MIMOD, differentiation of effort is focussed at the choice of mode strategy per population stratum. • Objectives of WP1 mixed-mode ASD: • Make an inventory of ASD implementations in ESS countries; • Structure the decisions/steps towards an mixed-mode ASD; • Illustrate using two case studies; MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  3. WP1 MIMOD survey • Findings from WP1 survey: • Mixed-mode ASD implemented only at Stat Netherlands; • ASD is a relatively unknown strategy to balance quality and costs. Eight countries indicated in survey they were unsure whether ASD is applied; • Potential reasons: • Implementation demands flexible case management system across modes; • Relatively weak available auxiliary data to stratify the population; • Mostly theoretical approach without many success stories; MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  4. ASD steps to implementation • Key ingredients of ASD: • Explicit quality and costs metrics; • Relevant auxiliary data; • Design features/interventions • In general: All possible elements of data collection strategy; • In MIMOD: Modes; • Optimization strategy, e.g. • Case prioritization; • Mathematical optimization; • Stopping rules based on quota MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  5. ASD steps to implementation – checklist • Identify priorities; • Identify major risks: • Consider risk of incomparability in time; • Consider risk of incomparability between subgroups; • Consider risk of budget overrun and heavy interviewer workloads in follow-up modes; • Define quality and cost indicators; • Consider nonresponse indicators; • Consider measurement error indicators; • Consider cost indicators; • Define decision rules from: • Trial-and-error; • Case prioritization; • Quota; • Mathematical optimization; • Modify the survey design and monitor the outcomes; • Develop a dashboard for survey errors; • Develop a dashboard for survey costs; • Compute estimates; • Document; MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  6. ASD case study – Health Survey/EHIS – priorities and risks • Key design feature: yes/no F2F follow-up to web nonrespondents • Main priorities: • Acceptable and similar response rates among relevant population subgroups • Sufficient precision on annual survey estimates • Costs satisfying a specified budget • Main risks: • Incomparability in time • Unpredictable CAPI workload due to varying monthly and annual web response rate • Incomparability between different population subgroups of interest MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  7. ASD case study – Health Survey/EHIS – quality and costs • Objective: • Maximize coefficient of variation (CV) of response propensities (combines R-indicator and response rate) • Response propensities modelled by age, income, urbanization, type of household, ethnicity • Constraints: • Aminimum annual total number of about 9500 respondents was requested • An upper limit of 8000 was imposed to the number of nonrespondents that are sent to CAPI, as a proxy for a budget constraint • An upper limit of 18000 persons was set to the sample size • TO DO: Inclusion of constraint on mode-specific measurement bias MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  8. ASD case study – Health Survey/EHIS – optimization Stratification based on classification tree of web response MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  9. ASD case study – Health Survey/EHIS – optimization Optimal allocation probabilities of web nonrespondents to F2F follow-up were determined based on mathematical optimization. Per monthallocationprobabilities are rescaled to guarantee a fixed F2F workload. CV uniform design = 0.158 CV adaptive design = 0.116 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  10. Concluding remarks • Adaptive mixed-mode survey design offers a flexible way to balance quality and budget • Holds true especially in sequential designs with more expensive (interviewer) modes as optional. • Further within mode differentiation (timing and number of calls/visits) is possible. • However: • Account of both representation and measurement is crucial; • A flexible case management system and monitoring is required; • Future: • May be combined with re-interview designs (WP2) • May be combined with sensor measurements/data (WP5) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

More Related