1 / 17

Stop the Madness: Use Quality Targets

Stop the Madness: Use Quality Targets. Laurie Reedman. Scope. Aspects of quality Timeliness and accuracy Mechanisms to manage quality Indicators and pre-set targets Survey processes Computer assisted interviewing Collection follow-up Manual processing.

tegan
Download Presentation

Stop the Madness: Use Quality Targets

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Stop the Madness:Use Quality Targets Laurie Reedman

  2. Scope • Aspects of quality • Timeliness and accuracy • Mechanisms to manage quality • Indicators and pre-set targets • Survey processes • Computer assisted interviewing • Collection follow-up • Manual processing Statistics Canada • Statistique Canada

  3. Statistics Canada’s dimensions of quality • Relevance • Accuracy • Timeliness • Accessibility • Interpretability • Coherence Statistics Canada • Statistique Canada

  4. Statistics Canada’s dimensions of quality • Relevance • Accuracy • Timeliness • Accessibility • Interpretability • Coherence Statistics Canada • Statistique Canada

  5. Statistics Canada’s dimensions of quality • Relevance • Accuracy • Timeliness • Accessibility • Interpretability • Coherence How can a survey manager manage both process and product quality of data collection and manual processing activities? Statistics Canada • Statistique Canada

  6. Interviewer Monitoring • Computer assisted interviewing • Monitor observes and grades samples of interviewer work • Frequency of monitoring sessions geared to attain desired average outgoing quality level Statistics Canada • Statistique Canada

  7. Responsive Collection Design (RCD) • An adaptive approach to survey data collection • Uses information prior to and during data collection to adjust the strategy for the remaining in-progress cases (Groves and Herringa, JRSS 2006) • Can use RCD to: • Control quality (response rate, representativeness) • Control cost (time and resources spent) Statistics Canada • Statistique Canada

  8. Responsive Collection Design (RCD) • RCD was piloted on 2 surveys at Statistics Canada (Laflamme and St-Jean, JSM 2011). • Three distinct phases during data collection • Early in collection – attempt all cases • Mid collection – increase response rates • Late collection – reduce variability of response rates between domains of interest • Key to success is changing from one phase to the next at the optimal time Statistics Canada • Statistique Canada

  9. Responsive Collection Design (RCD) • The turning point decisions are based on the comparison of quality indicators to pre-set target levels • Indicators are derived from paradata from current and previous collection activity • If targets are too high or too low the turning points will not be effective at improving quality • Targets need to reflect the priorities, for example to reduce costs, improve response rates, or optimize both simultaneously Statistics Canada • Statistique Canada

  10. Selective Editing and Top-Down Approach • Data editing is a quality assurance activity, not a data correction activity (John Kovar, 199?) • Goals: • Make data fit for use (not perfect) - effectiveness • Use as few resources as necessary - efficiency • Human resources to do telephone follow-up calls and manual analysis and data modification are costly • Managers need a mechanism to improve efficiency and effectiveness of manual processes without significantly impacting accuracy Statistics Canada • Statistique Canada

  11. Selective Editing and Top-Down Approach • Focus effort where it will do the most good (Hedlin, UNECE 2008) • Tackle efficiency and effectiveness from two angles: • Choose certain units or domains of units for further processing, cease processing of the rest • Arrange the units requiring further processing in priority order • Pro-actively control the impact on quality by basing turning points and priorities on comparisons of quality indicators to pre-set targets Statistics Canada • Statistique Canada

  12. Selective Editing • When to stop processing • Quality indicator could be mean squared error, coefficient of variation, response rate, calculated for key variables at cell or domain level • Targets need to be set carefully • If too high, might never be reached, end of processing will never be triggered, costs will not be reduced • If too low, resulting data might not be fit for use Statistics Canada • Statistique Canada

  13. Top-Down Approach • How to prioritize units needing more processing • Score function – to get a single rank incorporating several different criteria simultaneously • “Biggest” based on some size measure (prior knowledge) • “Biggest” based on a measure of impact (relative to what has already been collected) • Most outrageous errors (outlier detection) Statistics Canada • Statistique Canada

  14. An Example: Edit process • Canadian Census of Population edit and imputation process • 110 modules grouped into 43 processes • Underwent a “Quality Assurance Review” in 2013 (Reedman and Julien, FCSM 2013). • 65-70% of time was spent on manual data verification • Recommended increased use of automation, and pre-set quality thresholds to limit activity that amounts to “polishing the apple” Statistics Canada • Statistique Canada

  15. An Example: Edit process • Pro-active quality management could include: • Derive quality indicators for key variables, compare to pre-set targets, and direct satisfactory records onwards to the next processing step, while only retaining unsatisfactory records for appropriate intervention • Use a top-down prioritization method to further restrict manual intervention to only records having a significant impact • The effect on data accuracy and potential time (cost) savings could be estimated using Census 2011 data Statistics Canada • Statistique Canada

  16. Conclusions • Many sources of error in statistical processes • We looked at four ways to manage accuracy and timeliness in data collection and manual processing • Interviewer monitoring • Responsive Collection Design • Selective editing • Top-down prioritization • Using paradata • Feasibility and effectiveness demonstrated • Can be used separately or together Statistics Canada • Statistique Canada

  17. Thank-you! For more information, please contact: Laurie Reedman Statistics Canada Laurie.Reedman@statcan.gc.ca Statistics Canada • Statistique Canada

More Related