1 / 30

Quality Review of Key Indicators at Statistics Canada

Quality Review of Key Indicators at Statistics Canada. ICES-III, June 2007 Claude Julien and Don Royce. Outline. Quality Management at Statistics Canada Quality review process Recommendations Observations on the review process Conclusion. Quality Management at STC. 1997 Quality Audit.

barclay-rice
Download Presentation

Quality Review of Key Indicators at Statistics Canada

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quality Review of Key Indicators at Statistics Canada ICES-III, June 2007 Claude Julien and Don Royce

  2. Outline • Quality Management at Statistics Canada • Quality review process • Recommendations • Observations on the review process • Conclusion

  3. Quality Management at STC

  4. 1997 Quality Audit • Quality Assurance Framework (QAF) • Audit on 4 key surveys • Office of the Auditor-General recommended to examine a wider application of enhanced self-assessment

  5. Quality Assurance Framework (QAF) • Quality definition • Fitness-for-use: relevance, accuracy, timeliness, coherence, accessibility, interpretability • Management of each dimension • Policies, standards, guidelines and practices • Five main systems: users liaison, planning, methods and standards, dissemination and reporting (Brackstone 1999, Survey Methodology)

  6. Integrated Program Reports (IPR) • Quadrennial report and biennial follow-up • Quality, costs, management of HR and strategic direction • Self-assessment and user survey or consultation • Focus on quality of outputs • Review and feedback from Policy Committee

  7. Quality Review Process

  8. Initiating the review • Some serious errors in the preceding 18 months • International trade • Productivity estimates • Consumer Price Index • Questions from Finance, other key users and media • Policy Committee created Steering Committee of DGs to review quality assurance practices • Headed by DG-Methodology and assisted by head of Quality Secretariat

  9. Objectives of the review • Identify “best practices” that should be promoted to other programs • Identify specific weaknesses where action is needed

  10. Programs Reviewed • Monthly programs • Labour Force Survey • Retail Trade Survey • Manufacturing • International Trade • Consumer Price Index • GDP by industry • Quarterly programs • Income and Expenditure Accounts • Labour Productivity • Balance of Payments

  11. Challenges – Types of programs • Labour Force Survey • Retail Trade Survey • Manufacturing • International Trade • Consumer Price Index • GDP by industry • Income and Expenditure Accounts • Labour Productivity • Balance of Payments Traditional Administrative data Mixture of data Derived data Mixture of types

  12. Challenges – Program process • Process = complex series of steps • Set objectives, define population, create frame, select sample, collect data, etc. • Identify a reasonable number of recognizable steps • Focus on one or two

  13. Challenges – Aspects of a step • Design • Implementation • Execution • Evaluation

  14. Challenges – Dimensions of quality • Relevance • Accuracy • Timeliness • Coherence • Interpretability • Accessibility

  15. Mandate of the review • Review the practices during production (execution) to prevent the release of erroneous data (accuracy) • Particular focus on the certification and data dissemination of data • Review of Dissemination and Communications Program • Identify the underlying risk factors and their importance.

  16. Review Teams • Identify and “volunteer” 10 reviewers • 10 review teams • 3 reviewers + member of Steering Committee per team • Lead reviewer

  17. Organization Policy Committee Mandate Directors General Recruitment Directors Program Assistant Directors

  18. Introductory meeting • Program director, lead reviewer and head of QS • Introduce the review process • Introduce the program • Identify main steps of program process

  19. Main steps of program process • Preparation for certification • Data collection • Editing and transformation • Imputation and estimation • Certification • Data release • Post-release

  20. Gather information • Semi-structured questionnaire • Cover the topics of the mandate • Not to be completed • Interviews with review team and program director and managers • Available documentation • Meta-data • Program reports

  21. Assessment • Individual reports • Report by lead reviewer • Reviewed by Steering Committee • Vetted by program director • One-day debriefing session with all reviewers • Summary report • Report by Steering Committee • Presentation to Policy Committee

  22. Taking action • Disposition of recommendations by Policy Committee • Funding for corrective actions • Follow-up in Integrated Program Reports

  23. Recommendations

  24. Main recommendations • Program managers are well aware of the risks factors • The programs are well-positioned to identify the improvements • Some programs would benefit more than others from additional investment • Human resources concerns dominate all other risk factors.

  25. Main recommendations • The existence of a research and analysis capacity separate from the production operation is a key factor in assuring quality. • There are numerous “best practices” in all programs that can usefully be shared within the Agency. • Statistics Canada should develop a formal Quality Incident Response Plan • Proposals to further increase the timeliness of the programs should be regarded with extreme caution • Statistics Canada should establish an ongoing program of Quality Assurance Reviews

  26. Observations on the review process

  27. Aspects to bring forward • Clear mandate from Policy Committee to focus the review • Participation at the DG level to assure legitimacy • Strong cooperation from program directors and managers • Concurrent reviews to assure attention and team work • Knowledge and experience of reviewers • Composition of review teams to assure some standardization • Excellent learning, development and networking opportunity for reviewers • Overall positive experience for the reviewers • Collection approach and tools • One-day debriefing session

  28. Aspects to improve • Better assessment and reporting of level of various risk factors • Timeliness • 10 reviews in five months (Oct-Mar) • Reviewers spent around 15 days • More interviews than expected • Time to write, review and translate reports • More timely process and dissemination • More time to schedule meetings around production • More standard program reports

  29. Conclusion • The review was developed and carried out very quickly • The review was a success thanks to all participants • The review was not thorough; it complemented other key quality management mechanisms • Mandate to propose an on-going review program • Build on recent review • Criteria for selecting surveys • Tool box / Flexible approach • Part of learning and development program

  30. For more information, or to obtain a French copy of this presentation, please contact: Pour de plus amples informations ou pour obtenir une copie en français du document, veuillez contacter : Merci Claude Julien Email / Courriel: claude.julien@statcan.ca Phone number / Téléphone: 613-951-6937

More Related