Principles, criteria and methods Part 2

1. Quality management Principles, criteria and methods Part 2 Produced in Collaboration between World Bank Institute and the Development Data Group (DECDG)

2. European Statistics Code of Practice 15 principles: Continued: APPROPRIATE STATISTICAL PROCEDURES NON-EXCESSIVE BURDEN ON RESPONDENTS COST EFFECTIVENESS RELEVANCE ACCURACY AND RELIABILITY TIMELINESS AND PUNCTUALITY COHERENCE AND COMPARABILITY ACCESSIBILITY AND CLARITY • PROFESSIONAL INDEPENDENCE • MANDATE FOR DATA COLLECTION • ADEQUACY OF RESOURCES • QUALITY COMMITMENT • STATISTICAL CONFIDENTIALITY • IMPARTIALITY AND OBJECTIVITY • SOUND METHODOLOGY

3. ESCP questionnaire • Provides a tool for self-assessment of quality aspects • Designed in a way that most parts can be filled in centrally, e.g. by the quality manager of the organization • Follows the structure of the European Statistics Code of Practice - subdivided into 15 principles and 3-7 indicators for each principle • For each principle the questionnaire concludes with a follow-up part in which statistical authorities are requested to reflect upon improvement actions and a time frame

4. Quality management in practice • Quality management in the practice of statistical offices has two main aspects: • Assessing the quality of statistical outputs – users judge the quality of outputs on aspects relating to their use of the statistics. To meet user quality requirements, a framework for assessing the quality of outputs should be used • Managing quality in the ongoing production of statistics - a standard set of indicators for different parts of the production process should be used

5. Assessing the quality of outputs • The key assessment aspects for the users are: • Relevance • Accuracy • Coherence • Interpretability • Timeliness • Accessibility of data and metadata • To measure output quality for each of those, indicators can be used

6. Output quality indicators – accuracy/1 • Quality measures (e.g. sampling errors) and indicators (e.g. non-response rates) regularly produced and monitored for source data • Definitions of data consistent with both user and provider understanding • A revisions policy which balances the need to inform users of improved estimates with possible confusion from insignificant changes

7. Output quality indicators – accuracy/2 • Insignificant and consistent (size and direction) differences between preliminary and final estimates • Data rebased regularly • Data source samples redesigned or reselected regularly to maintain sample errors within quality standards

8. Output quality indicators – coherence/1 • All source data measured in accordance with standards (frame, statistical units, definitions, classifications, processes) • Data presented in a framework, along with other relevant data • Long term time series are available, with explanations or adjustments for breaks in the series • Input data from different sources are confronted and reconciled • Output data is consistent or reconcilable with other sources

9. Output quality indicators – coherence/2 • Concordances available to allow data to be related to previous classification versions or related classifications • Consistency between aggregates and components • Key classifications are maintained so that comparability over time is maintained

10. Output quality indicators – interpretability/1 • Seasonal and trend analysis and other adjustment techniques are undertaken to enhance the usefulness of the data and reduce problems of interpretation and comparisons • Preliminary or early estimates are clearly indicated as such and information provided on expected level of revisions • Major changes to the data from revisions, rebasing, etc. are published separately from, and where appropriate ahead of, the release of new information

11. Output quality indicators – interpretability/2 • All revisions are clearly marked and explained • Presentation standards used for tables and graphs • Information on methods, concepts, data sources, etc. Is up-to-date and readily available to users • Information on quality achieved is readily available

12. Output quality indicators - accessibility • Main findings are made widely available (e.g. through press releases, website, public libraries) • Information on data availability (published and unpublished) is readily known • Catalogues/directories are available for all fields of statistics • Information is available in formats and media required by users • First release of key data is made available in all media at the same time • Release dates are announced in advance • Information is affordable • New information and communication technology is being used to improve the presentation and accessibility of data

13. Managing qualityof processes • The following is necessary: • Quality standards specified and known to all process managers and staff • Processes and rules documented and the information readily accessible • Owners of the outcomes of each stage identified along with their responsibilities • Outputs at each stage defined along with standards and tolerances • Production and monitoring of indicators to ensure the standards are met and users can be informed of the quality achieved • A system for registering process problems and managing action taken • A system for change management

14. Monitoring the quality of surveys/1 • Examples of indicators to be monitored are: • Sample and non-sample errors • Response rates • Proportion of proxy interviews • Impact of births and deaths in business surveys • Impact of rotation

15. Monitoring the quality of surveys/2 • Outliers and imputation • Population change • Respondent load

16. Monitoring respondent management/1 • Key respondent management issues are: • Forms in source data collections tested with respondents • Good, up-to-date understanding of respondent information sources • Respondents able to provide requested information via preferred method • Help/support system available • Measures of load produced regularly

17. Monitoring respondent management/2 • Administrative data used wherever possible, for small firms tax data is normally a useful source. • Sound security of information • Confidentiality checking of releases

18. Quality support systems • Quality management is impossible without adequate support systems • Key tools are: • Expert service units for standards and techniques for reducing error (e.g. sampling methodologists, questionnaire designers, time series analysts) • Project management methodology to manage collections • Protocols with associated standards and guidelines for graphs, tables, time series presentation, revisions, releasing data with error, form design, etc • Standard frames, frameworks, definitions, questions, classifications, code files • Easy access to documentation • Peer reviews of the design of various aspects of a collection • Registers for logging problems and tracking their resolution

19. Quality management and leadership • High level management support and leadership is needed • Important principles are: • Output managers actively taking responsibility for the quality for their products • Information on quality regularly collected and used • Customer orientation • Documentation is encouraged • Good practices sought out and promoted • Extension of solutions/innovations to other processes • Regular reviews of performance of systems • Staff development and support

20. Managing staff quality/1 • The key to success in quality management is staff with the right skills • Important staff skills in this respect are: • It starts with a good understanding of the topic they report about. • That is subject matter knowledge. • Good understanding of key users and emerging new stakeholders • Good understanding of public policy directions and issues • Ensuring that information is produced at the right frequency to allow timely monitoring of changes. • Knowledge of the relevant processes and the methods they need to apply.

21. Managing staff quality/2 • Ensuring that information provides a useful estimate of what key users want to measure • Ensuring that questionnaires, definitions and classifications reflect contemporary needs and situations