Responsive Collection Design (RCD) for CATI Surveys and Total Survey Error (TSE)

1. Responsive Collection Design (RCD) for CATI Surveys and Total Survey Error (TSE) François Laflamme International Total Survey Error Workshop (ITSEW) Quebec, June 2011 Statistics Canada • Statistique Canada

2. Outline • Introduction • RCD surveys • RCD strategy • Active management for RCD • Monitoring and decision making tools • Highlights and lessons learned • RCD and TSE • Next Steps Statistics Canada • Statistique Canada

3. Responsive Collection Design (RCD) is an adaptive approach that uses the information available prior and during data collection to adjust collection strategy for the remaining in-progress cases Trade-off between quality, cost, productivity, responding potential of in-progress cases, survey mode and interaction between surveys Introduction Statistics Canada • Statistique Canada

4. Two experiment surveys with RD and control groups Households and the Environment Survey (HES 2009) Dwelling survey with a cross-sectional design Canadian Community Health Survey (CCHS 2009) sampling frame Survey of Labour and Income Dynamics (SLID 2010) Longitudinal survey (Complex survey design) One full RCD survey - SLID 2011 Embedded experiment for the first call Objectives: proof of concept, improve efficiency and quality RCD Surveys Statistics Canada • Statistique Canada

5. RCD Strategy • Planning phase • Analysis of previous data collection cycle • Data collection phases and strategies • RCD objectives, staffing plans and response propensity model for each survey • Sample validation • Active management tools and reports • New key indicators and communication plan • Control group to assess RCD impact Statistics Canada • Statistique Canada

6. RCD Strategy (cont’d) • Initial data collection phase • Use strategic improvement opportunities previously identified • New time slice strategy and intermediate cap • More likely to collecteasy cases • Monitor key indicators to identify start of RCD Phase 1 • Response rate, productivity, cost (proportion of budget spent) and responding potential of in-progress cases • By Regional Office (RO) Statistics Canada • Statistique Canada

7. RCD Strategy (cont’d) • RCD Phase 1 - Daily overnight job • Categorize and prioritize cases to improve overall response rates • Probability of completion (propensity) - logistic regression model (sampling frame and sequence of calls information) • Monitor key indicators to identify start of RCD Phase 2 • Representativity indicator and previous key indicators • By Regional Office (RO) • RCD Phase 2 - Daily overnight job • Prioritize cases to improve sample representativity • Priority to domains of interest with lower response rates Statistics Canada • Statistique Canada

8. RCD Strategy for HES and SLID For SLID 2010, a group called “High probability-Tracing” was used during RD phase 1 For SLID 2011, another group called “High probability-Refusal” was added during RD phase 1 The intermediate and global caps on calls were (20, 25) for HES and (30, 40) for SLID Statistics Canada • Statistique Canada

9. Active Management for RCD • Set of plans and tools to manage data collection while in progress • By Regional Office (RO) • Paradata and data sources used • Blaise Transaction file (BTH) (i.e. calls and contact information), interviewer payroll hours, budget and target figures, previous and current collection cycle information, response propensity model results • Key indicators • Used to identify when to start RCD Phase 1 and Phase 2 • No survey estimates monitoring so far Statistics Canada • Statistique Canada

10. When to initiate RCD Phases? • Decision based on survey progress in terms of response rate, productivity, proportion of budget spent (cost) and responding potential of in-progress sample Statistics Canada • Statistique Canada

11. RCD Dashboard - Example for RCD Phase 1 Dashboards are used to identify when to start both RCD phases to facilitate interpretation and objective decision-making RCD phase 1: 6 conditions, RCD phase 2: 7 conditions Yellow and red lights signal when many conditions are met 2014-06-07 Statistics Canada • Statistique Canada Statistics Canada • Statistique Canada 11

12. When to initiate RCD Phase 2? • Similar dashboard than RCD Phase 1 is used (7 conditions) • Condition 7: average response rate increase over the last 5 days • Representativity indicator is also used • Representativity indicator is a measure of variability of response rates between domains of interest • National versus regional objectives Statistics Canada • Statistique Canada

13. Other Active Management Tools for RCD - Examples • Interviewing progress, results and effort • Response and resolved rates, tracing and refusal conversion results and effort, refusal at the first contact, refusal and no contact rates • In-progress cases • Distribution of calls (for cap on calls monitoring), distribution of cases by Blaise group, browser use, intensive tracing, cases with high propensity but few calls • System time (effort) • By period of day, phase and group (RCD, CG) • Expected distribution by Blaise group between phases • Useful for staff planning • Several other ad hoc tools • Used to identify problems or emerging issues Statistics Canada • Statistique Canada

14. Active Management Challenges Large amount of information and reports available In the past not enough info, currently too much Need to concentrate on major issues (not on good to know info) Can spend a lot of time on something not broken Some analysis can wait at the end of collection Analysis and communication Often require an extra analytical step before the information is communicated Reports are not enough Real challenge is to analyse, summarize and communicate In the case of RCD, only the main reports were distributed Other reports were only used when required 2014-06-07 Statistics Canada • Statistique Canada Statistics Canada • Statistique Canada 14

15. Highlights and Lessons Learned • Higher overall response rate when RCD is used • Compared to previous survey cycle • RD group achieved same response rate with less effort (~2%) • Sample representativity generally improved • High probability and priority groups had positive impact • SLID 2011 • Better response rate and contact rate (over 2%) for cases for which the first call was forced to be in the same time slice of previous interview • Pilot test demonstrated the technical feasibility of RCD • Active Management and communication • Essential for any RCD • Need timely and accessible paradata • RCD is not a “magic” solution • Need to be used in conjunction with other initiatives Statistics Canada • Statistique Canada

16. RCD and TSE • RCD objectives in terms of TSE • Reduce variance and non-response bias • Currently, more likely the variance • But might depend on the non-response adjustment strategy • Did RCD introduce a potential non-response bias? • Active Monitoring of the non-response bias requires survey estimates during data collection - but can be done after • Did RCD change the responding propensity of cases? • Soft treatments - More meaningful grouping of cases • No incentives, no sub-sampling of respondents Statistics Canada • Statistique Canada

17. Next Steps • Implement RCD for other CATI surveys • Developed with a research rather than a production perspective • Improve current RCD strategy • Propensity models • Gradually phase-in of RCD phase1 • Integrate new conditions for decision making • Representativity, survey estimates, non-response bias • Include cost-efficiency objective • RCD for CAPI surveys • Potential benefits of RCD to improve cost-efficiency • Feasible study : communication flow, potential actions, concurrent surveys, operational and technical constraints. • RCD for multi-mode surveys • RCD theoretical framework Statistics Canada • Statistique Canada

18. For more information, please contact Pour plus d’information, veuillez contacter François Laflamme francois.laflamme@statcan.gc.ca Statistics Canada • Statistique Canada