Statistics – Investment in the future 2 September 14-15 2009, Prague, Czech Republic

1. The implementation of quality assurance frameworks for international and supranational organisations compiling statistics Statistics – Investment in the future 2 September 14-15 2009, Prague, Czech Republic Håkan Linden and Antonio Baigorri Quality and Classifications European Commission, Eurostat

2. Outline • Quality frameworks and TQM • Quality assurance frameworks for international organisations • Quality assurance methods and tools • The International Statistical Processes Assessment Checklist (ISPAC) • Conclusions • Outlook

3. The context of Total Quality Management • To have an encompassing approach with respect to quality work. • To implement the principles of institutional quality frameworks and in particular the principles related to statistical processes and outputs. • To improve the measurement, monitoring and management of data quality. • To coordinate ongoing quality initiatives (process descriptions, quality reports, evaluation activities etc.). • To build on existing quality work (standards, best practices etc.). • To promote a culture of systematic quality improvement work.

4. Quality Frameworks – how they relate Institutional frameworks, like the Principles Governing Statistical Activities, the European Statistics Code of Practice and the IMF Data Quality Assessment Framework, can be seen as general superstructures forming the necessary basis for all other measures an International organisation needs for improving quality at statistical output and product level.

5. Quality Assurance Frameworks • Quality assurance frameworks (or frameworks for statistics production) have the objective to establish, in a specific statistical organisation, a system of coordinated methods and tools guaranteeing the adherence to minimum requirements concerning the statistical processes and products. Similarly to institutional frameworks, this includes some kind of assessment. • Product/ output quality requirements are being explicitely documented. • Processes are defined and made known to all staff. • The correct implementation of the processes is monitored on a regular basis. • Users are being informed on the quality of the products and possible deficits. • A procedure is implemented that guarantees that the necessary improvement measures are being planned, implemented and evaluated.

6. User Standards requirements 3 - Conformity Labelling, Certification 2 - Evaluation Quality assessments 1 - Documentation and measurement User Process Quality reports satisfaction and indicators descriptions surveys Quality improvements Statistical Production User perception products processes Quality Assurance Methods and Tools

7. Relevance • Accuracy (and reliability) • Timeliness • Punctuality • Accessibility • Clarity/ interpretability • Coherence/ consistency • Comparability • ESMS • SDDS • IMF Data ROSC user surveys • Eurostat user satisfaction surveys (2007 and 2009) The assessment methods and tools ESS Quality & Performance Indicators • Rate of available statistics • Coefficient of variation • Rate of overcoverage • Edit failure rate • Unit response rate • Item response rate • Imputation rate • Number of mistakes • Average size of revisions • Time lag ref. period and first results • Time lag ref. period and final results • Punctuality of publication • Number of subscriptions of reports • Number of accesses to databases • Rate of completeness of metadata • Lengths of comparable time series • Asymmetries for statistics mirror flows • User satisfaction index • Length of time since most recent user survey • Annual operational costs • Annual respondent burden • Documentation and measurement - Quality reports (producer quality reports and user quality reports) - Quality and Performance Indicators - User satisfaction surveys - Process descriptions • Evaluation - self assessments of all production processes - quality reviews for key statistical outputs • Conformity - a process for labelling of key international statistics (for the future…) The Generic Statistical Business Process Model (v.4) Specify needs Design Build Collect Process Analyse Disseminate Archive Evaluate International Statistical Processes Assessment Checklist • OECD • EUROSTAT

8. Quality and metadata • Collection and sharing of metadata - SDMX technical standards - SDMX Content oriented guidelines (incl. Cross domains concepts) • Dissemination of metadata on quality - Special Data Dissemination Standards (SDDS) - Euro-SDMX Metadata Structure (ESMS) • Assessment and monitoring of metadata - Availability of up-to-date information - Quality of the information provided • Integrated information on quality assessment - SDMX tools

9. How to apply quality assessment methods and tools • The office-wide management approach • Institutional preconditions (procedures and legislations) • Assessment methods already in use • Relevance – size and periodicity • Relevance – importance and specific legal frameworks

10. Why do quality assessments? • The perception of the statistical product by the user. • The characteristics of the statistical product (or key statistical outputs) • The characteristics of the statistical production process.

11. Follow-up (10) User needs (3) Data collection (4) Validation Country level (5) International level (6) Confidentiality (7) Documentation (8) Dissemination (9) Conceptual framework (2) IT conditions (11) – Management, planning and legislation (12) – Staff, work conditions and competence (13) Relationship between process and output quality RELEVANCE ACCURACY ACCESSIBILITY/ CLARITY TIMELINESS/ PUNCTUALITY COMPARABILITY COHERENCE Source: The International Statistical Processes Assessment Checklist [CCSA, 2009]

12. The International Statistical Processes Assessment Checklist (ISPAC) • To check if the necessary quality assurance procedures are in place • To identify the strengths and weaknesses of the process • To help identify possible improvements to the process • To record good practices

13. The International Statistical Processes Assessment Checklist (ISPAC) Main outputs: • Process assessment checklist • Summary report • Good practices

14. ISPAC - principles for implementation • Minimise burden for production domains - test the approach in advance - provide support - flexibility • Build on existing information - process analysis - metadata on quality (quality reports etc.) • Profit from synergies with other horizontal activities - evaluation function requirements - cost/ benefit analysis - input for management programming

15. Keys for success in implementing quality assessments • Top management commitment • Middle management acceptance • Sound communication • Long term perspectives • Implementation and fine-tuning in pilot projects • Standardised use of methods • Clear responsibilities and ownership • Sufficient resources allocated for the assessments

16. Conclusions • Each international organisation should have a quality assurance framework in place. • The framework and the applied quality principles should be made explicit. • A quality assurance framework needs to be compatible with the general quality management model and office-wide procedures and rules. • It should be built into the organisational structure. • It contributes to increased awareness of quality concepts and promotes best practices. • It provides a mechanism for reengineering and quality improvements • It should always acknowledge performance/ cost. • Convergence of quality assurance frameworks by applying common concepts, standards, methods and tools (both content oriented and technical).

17. Outlook • Users needs for data about complex phenomena • Process integration (IT architecture and tools) • Standardisation tools (CBM’s, guidelines, standards etc.) • Quality assessment in integrated production models • Management of metadata and other infrastructure elements • Simplification of legislation/ frameworks for statistics production