Maria João Zilhão Planing and Quality Control Unit

1. Maria João Zilhão Planing and Quality Control Unit « Statistical Auditing and Self-Assessment Benefits for Quality Improvement « Q 2008 / Rome July 2008

2. Concepts …. Self-Assessment Self-assessment is a comprehensive, systematic and regular review of an organization’s activities and results referenced against a model/framework An audit is a systematic, independent and documented process for obtaining audit evidence and evaluating it objectively to determine the extent to which the audit criteria are fulfilled Audit

3. Recommendation nº 16 Recommendation nº 15 The Importance of Auditing and Self-assessment in statistics The Leadership Group on Quality A generic checklist should be developed for a simple self-assessment programme for survey managers DESAP The methods for auditing on different levels and for diferent purposes such as internal, external ,.., should be reiviewd and recommendations provided Handbook on experiences on auditing European Statistics Code of Practice 4.2 4.5 Principle 4 – Quality Commitment

4. The DatQAM map(Handbook on Data Quality assessment Methods and Tools Eurostat / 2007) External Environment of DatQAM User Requirements Standards Labelling Certification (ISO 20252) Self-Assessment Audits Process Variables Quality Indicators Quality Reports User Surveys Precoditions of DatQAM Measurement of Processes and Products

5. ISO Norms: 9000:2000…Implementation of Quality Management Systems 19011 - .. Auditing… 10013 - …Documentation systems References for auditing

6. References for self-assessment • EFQM Excellence Model (European Foundation for Quality Management) • CAF – Common Assessment Framework (EIPA - European Institution for Public Administration) • DESAP – Checklist for survey Managers (Eurostat)

7. Top Management support The results of auditing and self-assessments must be consequent Proper preparation! A Clear communication and involvement of the staff Documentation System Preconditions

8. P D A C Quality Management System at Statistics Portugal Quality Documentation System ISO 19011 ISO 10013 Departments Documenting Processes Departments Departments Suggestions / Complaints Internal Audits Customer Satisfaction Surveys External Audits Performance indicators Quality Management Unit COP – Principle 4

9. Process design Definition of different phases of the process Definition of responsibilities Definition of minimum documentation Handbooks are produced by pluridisciplinary teams Documentation System Documenting processes ISO 10013 Procedures handbooks Documentation on survey processes

10. Statistical Production Handbook Procedures Phases II. Operation III. Dissemination IV. Evaluation I. Conception Processes Processes Process Process II.1. Planing and preparation of data collection I.1. Viability Study III.1. Dissemination IV.1. Evaluation I.2. Methodological Study II.2. Data collection II.3. Data treatment and analysis Audits I.3. Technical approval Note: Each Process is composed by sub-processes and tasks

11. P D A C Internal Quality Audits at Statistics Portugal (I) Objectives To verify the degree of procedures implementation (handbooks procedures) To Identify improvement opportunities To check the internal articulation/communication of units (survey related) To Improve survey documentation

12. Internal Quality Audits at Statistics Portugal (II) Statistical Production Handbook Procedures II. Operation III. Dissemination IV. Evaluation I. Conception II.1. Planing and preparation of data collection I.1. Viability Study III.1. Dissemination IV.1. Evaluation I.2. Methodological Study II.2. Data collection II.3. Data treatment and analysis I.3. Technical approval «

13. Internal Quality Audits at Statistics Portugal – Preparation (III) Training of 14 Auditors (Statisticians) (ISO 19011) Discussion Top Management Support Exchange of Information / Strong Communication about auditing, through Top Management, and the Quality Unit Internal Procedure on auditing (Clear rules!)

14. Internal Quality Audits at Statistics Portugal –(IV) Principal rules Top Management defines the Quality/Statistical Audits’ annual plan No one should audit its own activity One Audit team needs 2 or 3 auditors Auditors produce Audit Plans and reports according to Procedures Improvement actions must be carried out by the audited departments Improvement actions must be published and followed by the audit teams

15. Auditors are Peers (from other statistical Offices) Basic rules are the same as for internal auditing Two visits (+/- one week each) Contacts between auditors and audited teams in the mid period Auditors work recommendations with audited teams Results are presented to the board Action plan is made by audited teams and presented to the board External Quality Audits at Statistics Portugal

16. Benefits for Quality Improvement Some NSI’s have experienced Quality Auditing! The identification of improvement opportunities in the audited process The implementation of corrective / preventive actions The redefinition of procedures Auditing involves people in the quality program (auditors and audited teams) Good Recommendations - A look from the outside! Identification of good practices that can be spread in the organization Self-assessment and auditing can be combined together A time to re-think and improve! A Powerfull Tool! • BUT!!!! • -Time consuming activity • May face resistances • May be dificult to start • - Better results in the long run if systematically implemented

17. THANK YOU! Maria João Zilhão Statistics Portugal Planing and Quality Control Unit mjoao.zilhao@ine.pt « «