Quality Reporting at SORS – Experiences and Future Perspectives

1. Quality Reporting at SORS – Experiences and Future Perspectives Rudi Seljak, Tina Ostrež Statistical Office of the Republic of Slovenia

2. The summary of the presentation • History of quality reporting at SORS • Feedback from the producers of quality reports • Expanding of the area of quality reporting • Future plans • Conclusions

3. Quality reporting at SORS • The systematic work on quality reporting at SORS began in 2003. • On the basis of the “European” methodological documents our own quality reporting framework was developed. • Cornerstones of the framework: • The standard template with prescribed structure for quality reporting • The list of standard quality indicators • The first quality reports were prepared in 2004 and 2005 → just for internal use. • In 2006 the first publicly available reports were disseminated.

4. Quality reporting at SORS cont’d • Two types of Quality reports are disseminated: Standard quality reports • Produced every five years • Exhaustive document → covering all aspects of quality • Available only in Slovene language http://www.stat.si/metodologija_porocila-standardna.asp Annual quality reports • Produced every year • Short document → mostly based on quality indicators • Translated also to English http://www.stat.si/eng/metodologija_porocila-kakovost.asp

5. Procedure of preparation • At the beginning of the year the domains, for which the reports should be prepared, are identified. • The workshop with the subject-matter statisticians is organized: • The theoretical and practical views of quality assessment are presented • The timetable is set up • During the preparation of the reports further assistance is provided by the coordinators. • When the report is written, it is reviewed by the coordinators before the final version is accomplished.

6. Survey among producers • A small-scale survey was conducted among the persons involved in QR preparation in order to: • get the general feedback (burden, usefulness,…) • get the useful information for future planning • The survey took place in December 2009. • From around 40 standard quality reports prepared by that time, we obtained answers for 32 domains.

7. Main findings of the survey • The perceived burden due to the preparation of standard quality reports is high, but varies a lot: 8-200 hours (average: 46 hours). • For annual quality reports: 1 – 50 hours (avarage:17 hours) • Double work in the case where the QR from Eurostat is demanded was pointed out → degree of comparability of two concepts sometimes found very low • Also some positive feedback were provided: • The organization of work was found very good • The preparation has forced them to think about their own work in a new way • Deeper insight into the data they are preparing

8. Expanding the area of quality reporting • The concepts set up in 2003 mainly targeted the “classical” statistical surveys. • Use of registers and different administrative sources is very popular in practice of SORS → need for adjusted concepts clearly pointed out. • The template as well as the list of quality indicators should be adjusted. • Also the need for adjustments for some specific domains (price statistics, national accounts) was pointed out.

9. Quality assessment and administrative data • Some distinctive characteristics of quality assessment when admin data are used: • The data were not collected for the statistical purposes → suitability for statistical purpose must carefully be studied → relevance dimension becomes a crucial part of the assessment. • The administrative data are determined by the legal world → danger of describing the world as it is “de-iure” instead of as it is “de-facto” → consideration of this problem should be put in the quality report. • The data collection process is dislocated from the statistical process → often wrongly assumed that data are free of measurement (as well as non-response, coverage) errors. To assess these errors close cooperation with the data provider is needed.

10. Plans for expansion of the quality framework • The adjusted quality concepts for the surveys which use administrative data or combine different sources: • Adjusted (more flexible) template for quality reporting • Adjusted list of standard quality indicators • Planed to be developed in 2010 • Quality reporting concepts for registers and large administrative data sources (shared by several surveys): • New template should be developed • The standard list of quality indicators will probably essentially be adjusted • Planed to be developed in 2011 and implemented in 2012

11. More plans for the future • The quality reports should be accomplished for around 50 additional statistical surveys. • The further harmonization with the Eurostat concepts should be carried on. • Improvements in the availability of the quality reports on the website are planned. • The completion of the database of quality indicators and its inclusion in the process of quality reports preparation is one of the main tasks in the following years.

12. Conclusions • SORS publicly disseminates quality reports from 2006 on. • Two types of reports are produced: standard quality reports and annual quality reports. • The survey among the producers of the reports showed that the perceived burden of the preparation is quite high but also some positive consequences were pointed out. • The main task for the future is to adjust the concepts for all different kind of surveys and to ease the burden of the preparation.