A Theoretical Framework for Adaptive Collection Designs

1. A Theoretical Framework for Adaptive Collection Designs Jean-François Beaumont, Statistics Canada David Haziza, Université de Montréal International Total Survey Error Workshop Québec, June 19-22, 2011

2. Selected literature review Framework Definition of the problem Choice of quality indicator and cost function Mathematical formulation of the problem Solution and discussion Conclusion Overview

3. Responsive designs: Use paradata to guide changes in the features of data collection in order to achieve higher quality estimates per unit cost Paradata: Data about data collection process Examples of features: mode of data collection, use of incentives , … Need to define quality and determine quality indicators Two main concepts: phase and phase capacity Literature review: Groves & Heeringa (2006, JRSS, Series A)

4. Phase: Period of data collection during which the same set of methods is used Phase 1: gather information about design features Phases 2+: alter features (e.g., subsampling of nonrespondents, larger incentives, …) A phase is continued until its phase capacity is reached Judged by the stability of an indicator as the phase matures Literature review: Groves & Heeringa (2006, JRSS, Series A)

5. Goal: determine an indicator of nonresponse bias as an alternative to response rates Proposed a quality indicator, called R-indicator: Population standard deviation must be estimated Response probabilities, , must be estimated using some model An issue: indicator depends on the proper choice of model (choice of auxiliary variables) Literature review: Schouten, Cobben & Bethlehem (2009, SM)

6. Another issue: indicator does not depend on the variables of interest but nonresponse bias does Maximal bias of : is the unadjusted estimator of the population mean: Two limitations of maximal bias (and R-indicator): unadjusted estimator is rarely used in practice depends on proper specification of Literature review: Schouten, Cobben & Bethlehem (2009, SM)

7. Goal: Reduce nonresponse bias through case prioritization Suggest targeting individuals with lower estimated response probabilities For instance, give them larger incentives or give interviewer incentives Their approach is basically equivalent to trying to increase the R-indicator (or achieving a more balanced sample) Recommend using auxiliary variables that are associated with the variables of interest Literature review: Peytchev, Riley, Rosen, Murphy & Lindblad (2010, SRM)

8. Development and implementation of responsive designs for CATI surveys at Statistics Canada Planning phase: before data collection starts (determination of strategies, analyses of previous data, …) Initial collection phase: evaluate different indicators to determine when the next phase should start Two Responsive Designs (RD) phases Literature review: Laflamme & Karaganis (2010, ECQ)

9. RD phase 1: prioritize cases (based on paradata or other information) with the objective of improving response rates increase the number of respondents (desirable) RD phase 2: prioritize cases with the objective of reducing the variability of response rates between domains of interest (increasing R-indicator) likely reduce the variability of weight adjustments (desirable) Literature review: Laflamme & Karaganis (2010, EQC)

10. First paper to propose a theoretical framework for adaptive survey designs Suggest: Maximizing quality for a given cost; or Minimizing cost for a given quality Requires a quality indicator (e.g., overall response rate, R-indicator, Maximal bias, …) Which one to use? Literature review: Schouten, Calinescu & Luiten (2011, Stat. Netherlands)

11. Adaptive collection design: Any procedure of calls prioritization or resources allocation that is dynamic as data collection progresses Use paradata (or other information) to adapt itself to what is observed during data collection Focus on calls prioritization Our objective: Maximize quality for a given cost Context: CATI surveys Definition of the problem

12. Focus of the literature: Find collection designs that reduce nonresponse bias (or maximize R-indicator) of an unadjusted estimator We think the focus should not be on nonresponse bias. Why? Any bias that can be removed at the collection stage can also be removed at the estimation stage We suggest reducing nonresponse variance of an estimator adjusted for nonresponse Choice of quality indicator

13. Suppose we want to estimate the total: Assuming that nonresponse is uniform within cells, an asymptotically unbiased estimator is: Quality indicator: The nonresponse variance Quality indicator

14. Overall cost: Overall cost

15. Expected overall cost: Expected overall cost

16. Objective: Find that minimizes the nonresponse variance subject to a fixed expected overall cost, Solution: Note: Equivalent to maximizing the R-indicator only in a very special scenario Mathematical formulation

17. Find the effort (number of attempts) necessary to achieve the target response probability Procedure: Select cases to be interviewed with probability proportional to the effort Issues: 1) Avoid small estimated to avoid an unduly large effort 2) Might want to ensure that a certain time has elapsed between two consecutive calls Implementation

18. Graph of variance vs cost Minimum nonresponse variance Expected overall cost

19. Solution of the optimization problem is found before data collection starts May be a good idea to revise the solution periodically (e.g., daily) Some parameters might need to be modified Update remaining budget and expected overall cost The revised optimization problem is similar to the initial one Revised solution

20. Solution (same as before): Revised target response probability: Effort: Revised solution Could be negative

21. Next steps: Simulation study Adapt the theory for practical applications Test in a real production environment Which quality indicator? Nonresponse variance? Others? Reduction of nonresponse bias: subsampling of nonrespondents Our approach could be used within the subsample Conclusion

22. Thanks - Merci • Pour plus d’information, veuillez contacter : • For more information, please contact: Jean-François Beaumont (Jean-Francois.Beaumont@statcan.gc.ca) David Haziza (David.Haziza@umontreal.ca)