Stop the Madness: Use Quality Targets

1. Stop the Madness:Use Quality Targets Laurie Reedman

2. Scope • Aspects of quality • Timeliness and accuracy • Mechanisms to manage quality • Indicators and pre-set targets • Survey processes • Computer assisted interviewing • Collection follow-up • Manual processing Statistics Canada • Statistique Canada

3. Statistics Canada’s dimensions of quality • Relevance • Accuracy • Timeliness • Accessibility • Interpretability • Coherence Statistics Canada • Statistique Canada

4. Statistics Canada’s dimensions of quality • Relevance • Accuracy • Timeliness • Accessibility • Interpretability • Coherence Statistics Canada • Statistique Canada

5. Statistics Canada’s dimensions of quality • Relevance • Accuracy • Timeliness • Accessibility • Interpretability • Coherence How can a survey manager manage both process and product quality of data collection and manual processing activities? Statistics Canada • Statistique Canada

6. Interviewer Monitoring • Computer assisted interviewing • Monitor observes and grades samples of interviewer work • Frequency of monitoring sessions geared to attain desired average outgoing quality level Statistics Canada • Statistique Canada

7. Responsive Collection Design (RCD) • An adaptive approach to survey data collection • Uses information prior to and during data collection to adjust the strategy for the remaining in-progress cases (Groves and Herringa, JRSS 2006) • Can use RCD to: • Control quality (response rate, representativeness) • Control cost (time and resources spent) Statistics Canada • Statistique Canada

8. Responsive Collection Design (RCD) • RCD was piloted on 2 surveys at Statistics Canada (Laflamme and St-Jean, JSM 2011). • Three distinct phases during data collection • Early in collection – attempt all cases • Mid collection – increase response rates • Late collection – reduce variability of response rates between domains of interest • Key to success is changing from one phase to the next at the optimal time Statistics Canada • Statistique Canada

9. Responsive Collection Design (RCD) • The turning point decisions are based on the comparison of quality indicators to pre-set target levels • Indicators are derived from paradata from current and previous collection activity • If targets are too high or too low the turning points will not be effective at improving quality • Targets need to reflect the priorities, for example to reduce costs, improve response rates, or optimize both simultaneously Statistics Canada • Statistique Canada

10. Selective Editing and Top-Down Approach • Data editing is a quality assurance activity, not a data correction activity (John Kovar, 199?) • Goals: • Make data fit for use (not perfect) - effectiveness • Use as few resources as necessary - efficiency • Human resources to do telephone follow-up calls and manual analysis and data modification are costly • Managers need a mechanism to improve efficiency and effectiveness of manual processes without significantly impacting accuracy Statistics Canada • Statistique Canada

11. Selective Editing and Top-Down Approach • Focus effort where it will do the most good (Hedlin, UNECE 2008) • Tackle efficiency and effectiveness from two angles: • Choose certain units or domains of units for further processing, cease processing of the rest • Arrange the units requiring further processing in priority order • Pro-actively control the impact on quality by basing turning points and priorities on comparisons of quality indicators to pre-set targets Statistics Canada • Statistique Canada

12. Selective Editing • When to stop processing • Quality indicator could be mean squared error, coefficient of variation, response rate, calculated for key variables at cell or domain level • Targets need to be set carefully • If too high, might never be reached, end of processing will never be triggered, costs will not be reduced • If too low, resulting data might not be fit for use Statistics Canada • Statistique Canada

13. Top-Down Approach • How to prioritize units needing more processing • Score function – to get a single rank incorporating several different criteria simultaneously • “Biggest” based on some size measure (prior knowledge) • “Biggest” based on a measure of impact (relative to what has already been collected) • Most outrageous errors (outlier detection) Statistics Canada • Statistique Canada

14. An Example: Edit process • Canadian Census of Population edit and imputation process • 110 modules grouped into 43 processes • Underwent a “Quality Assurance Review” in 2013 (Reedman and Julien, FCSM 2013). • 65-70% of time was spent on manual data verification • Recommended increased use of automation, and pre-set quality thresholds to limit activity that amounts to “polishing the apple” Statistics Canada • Statistique Canada

15. An Example: Edit process • Pro-active quality management could include: • Derive quality indicators for key variables, compare to pre-set targets, and direct satisfactory records onwards to the next processing step, while only retaining unsatisfactory records for appropriate intervention • Use a top-down prioritization method to further restrict manual intervention to only records having a significant impact • The effect on data accuracy and potential time (cost) savings could be estimated using Census 2011 data Statistics Canada • Statistique Canada

16. Conclusions • Many sources of error in statistical processes • We looked at four ways to manage accuracy and timeliness in data collection and manual processing • Interviewer monitoring • Responsive Collection Design • Selective editing • Top-down prioritization • Using paradata • Feasibility and effectiveness demonstrated • Can be used separately or together Statistics Canada • Statistique Canada

17. Thank-you! For more information, please contact: Laurie Reedman Statistics Canada Laurie.Reedman@statcan.gc.ca Statistics Canada • Statistique Canada