Marina Signore Head of Service “Audit for Quality Istat

1. Assessing Quality through Auditing and Self-Assessment Signore M. , Carbini R., D’Orazio M., Brancato G., Simeoni G. Marina Signore Head of Service “Audit for Quality Istat

2. Auditing and Self-assessment Presentation • Describe the new procedure for assessing quality launched by Istat and how it fits into Istat Quality Policy • The new procedure was launched in October 2009 and is now in the testing phase

3. DatQAM map External Environment Users Requirements Standards Labelling Certification (ISO 20252) Audit Self-Assessment Tools Process Variables Quality Reports User Surveys Quality indicators Measurements Documentation Preconditions

4. Auditing and Self-assessment Istat Quality Policy Harmonized with international quality frameworks European Statistics Code of Practice (since 2005) Eurostat Quality dimensions Centralized Structure for Quality Assurance (management and assessment)

5. Auditing and Self-assessment Istat Framework for Quality Improvement Actions Communication on Quality to Users Actions Internal Audit Self-Assessment Assessment of Standard Quality Indicators Assessment Survey Assurance- Quality Control Systems Measurements Quality Guidelines Documentation system Standard Quality Indicators Preconditions

6. Auditing and Self-assessment Quality Assessment • Current surveys • Analyses of SQI managed in SIDI • In-depth analyses of survey process • self-assessment • internal auditing • New Surveys • Procedure for quality labeling • developed and tested on 2 surveys • adherence to few principles (relevance, questionnaire design and test, response burden reduction, quality assurance system) • not yet launched

7. Auditing and Self-assessment • Quality Assessment for current processes • Analyses of SQI managed in SIDI • analyses of trends (response rates, timeliness,…) • evaluation of Istat policies • reports prepared by central service for Quality Regular evaluation for most statistical processes but not a specific tool for promoting quality improvements

8. Auditing and Self-assessment Quality Assessment for current processes Development (October 2009) of a strategy that combines audit and self-assessment • way for highlighting strengths and weaknesses • tool for improving weaknesses • tool for collecting and disseminating good practices Concrete way of improving quality

9. Auditing and Self-assessment Quality Assessment: main features • The audits are carried out by internal auditors (face-to-face interview) • The output of the assessment process consists in the identification of improvement actions, resources and timetable • The improvement actions resulting from self-assessment are reviewed by a team of reviewers • Additional resources or methodological support in order to implement the improvement actions • User consultation on the improvement actions and for setting priorities • Establishment of a Quality Committee in order to coordinate and sponsor the assessment activity

10. Auditing and Self-assessment Quality Committee • Established in February 2010 • Composition • Chaired by the Head of Department of Statistical Production • Co-chaired by the Head of Service “Audit for Quality” • Members: Head of services

11. Auditing and Self-assessment Quality Committee • Mandate • To coordinate audit and self-assessment activity: • Evaluation of final reports • Approval of improvement actions • Monitoring of improvement actions • To develop procedures for monitoring the quality of statistical processes • To coordinate and sponsor quality assessment in the ISS • To promote the adoption of the CoP in the ISS

12. Auditing and Self-assessment The tools • Quality guidelines • Audit questionnaire • Self-assessment questionnaire (electronic version) • Operative manual

13. Auditing and Self-assessment Quality Guidelines Reference guide to design, carry out and control a statistical process • Part I: Process quality • Survey purposes • Survey design • Registers • Data collection • Data processing • Data dissemination and documentation Part II: Product quality Components of Eurostat Quality definition

14. Auditing and Self-assessment Quality Guidelines For each section • Definition of the Principle to be followed • Text describing what should be done in order to fulfill the principle • Quite general descriptions and references to technical documents • Light document less than 50 pages

15. Auditing and Self-assessment Audit and Self-assessment questionnaires Structure • Part I: Process quality • Part II: Product quality • Part III: Process management and quality management For each section • The Principle to be followed is stated • Some questions in order to evaluate the degree of compliance to the Principle • Slight differences in the two questionnaires • Input: DESAP checklist and OECD questionnaire

16. Auditing and Self-assessment The testing phase • To test the tools and the procedure • To evaluate the impact and sustainability Features • on a voluntary basis • Audit: 2 processes(2 teams of 3 experts each) • Self-Assessment: 5 processes(3 teams of 3 reviewers each)

17. Auditing and Self-assessment Timetable of the new procedure • Development of the procedure and tools (October - December 2009) • Internal discussion on the tools (January 2010) • Testing phase (February- June 2010)

18. Auditing and Self-assessment Final remarks • Great opportunity for improving the quality of Istat statistical processes • Analyses of the testing phase but first results seem to be very positive • Further developments will concern processes based on administrative sources and secondary studies (e.g. National Accounts)