European Conference on Quality in Official Statistics (Q2010) 4-6 May 2010, Helsinki, Finland

1. Consulting the users on quality documentation supply Brancato G., Carbini R., Murgia M., Simeoni G. Istat, Italian National Statistical Institute Presenter: Giovanna Brancato European Conference on Quality in Official Statistics (Q2010) 4-6 May 2010, Helsinki, Finland

2. Consulting the users on metadata and quality documentation supply Consulting the users on quality documentation supply Background Istat project, financed in the framework of Eurostat grants on quality management and evaluation Objectives To evaluate users’ needs and satisfaction with regard to some metadata and quality documentation tools, developed at Istat and coherent with Eurostat metadata standards. The tools are: a) SIQual (Information System on Quality) b) Process report c) Quality report • Methods • a): Web survey to quality managers from European Statistical Agencies • b) and c): National users’ consultations by means of focus groups • Istat team work • Communication sector: A.M. Tononi • Survey managers: A. Mancini, S. Montecolle • Data capturing and web surveys: R. Balestrino, M. Degortes, • M. Landriscina, A. Nunnari

3. Istat Metadata and Quality documentation tools (1) Consulting the users on quality documentation supply • SIQual • Reference metadata: content, process characteristics, methodologies • On line data: publications, databanks • Documentation: regulations, instructions, manuals questionnaires • Short descriptions, standardised language • Italian and English • Standard quality indicators (for internal use only)

4. Istat Metadata and Quality documentation tools (2) Consulting the users on quality documentation supply • Process report • 3-5 page long • Process description • 1) Introduction • 2) Informative needs • 3) Survey design • 4) Data collection • 5) Data treatment • 6) Estimation and validation • 7) Documentation • Highly standardised and integrated to the quality documentation system

5. Istat Metadata and Quality documentation tools (3) Consulting the users on quality documentation supply • Quality report • 2-3 page long • Product quality-oriented • Following Eurostat quality dimensions: Relevance, Accuracy and reliability, Timeliness, Accessibility and clarity, Comparability and Coherence • Incorporating a minimum set of standard quality indicators (coefficient of variation, unit response rate, net imputation rate, modification rate, mean and mean absolute revision index, timeliness, length of comparable time series) • Provided with a Reading guide useful for quality definitions and quality indicators’ formulas

6. Eurostat quality documentation standards Consulting the users on quality documentation supply • Quality definition • Quality dimensions • Standard quality indicators and reports (EHQR*) • Euro SDMX Metadata Structure (ESMS) • * ESS Standards and Handbook for Quality Reports

7. Euro SDMX Metadata Structure, ESMS (1) Consulting the users on quality documentation supply ESMS Metadata items

8. Euro SDMX Metadata Structure, ESMS (2) Consulting the users on quality documentation supply ESMS Quality Indicators

9. Euro SDMX Metadata Structure, ESMS (3) Consulting the users on quality documentation supply ESMS Quality Indicators (cont.)

10. Quality report Process report Istat metadata mapping with ESMS Consulting the users on quality documentation supply SIQual - METADATA

11. Istat quality indicators mapping with ESMS Consulting the users on quality documentation supply Quality Indicators *Under study for a next implementation

12. Istat quality indicators mapping with ESMS Consulting the users on quality documentation supply Quality Indicators (cont.) *Under study for a next implementation

13. Evaluation methods for SIQual Consulting the users on quality documentation supply WEB SURVEY TO: Users/producers: quality managers members of Eurostat Working Group on Quality QUESTIONNAIRE SIQual from a general point of view: - usefulness - redundancy - lack of information - layout - friendliness Specific content areas. Clarity, easiness, and comprehensibility of: - homepage - 3 search criteria pages - single process description - help points and glossary SOFTWARE XsiMpLe, a prototype of software for computer assisted data collection, developed at Istat and based on open source technologies

14. Evaluation methods for the process and quality reports Consulting the users on quality documentation supply FOCUS GROUPS - Theory Discussion among a small group (6-10) of target population members. Objectives: to explore some pre-identified critical issues Methods: - guided by a moderator - lasting a couple of hours - several focus groups for different sub-populations. Critical issues: role of the moderator (create relaxed atmosphere, maintain the discussion within the framework of interest, …).

15. Evaluation methods for the process and quality reports Consulting the users on quality documentation supply • APPLICATION • Selection of two Istat important statistical processes: • Monthly survey on industrial production index • Multipurpose survey: aspects of daily life • Identification of the 6 users for each process to invite to the focus groups: • 1° focus group participants: two economic journalists, • four economic analysts. • 2° focus group participants: one social sector journalist, • three academic representatives, • two analysts from research agencies • Sending of the material in advance • Organisation of 1 focus group for each process, lasting from 11:00 a.m. to 5:00 p.m. • Audio-recording of the meetings • Conditions different from the standard application field

16. Preliminary results Consulting the users on quality documentation supply • WEB SURVEY ON SIQual • Launched on mid-April  First results available in June • FIRST FOCUS GROUP • General satisfaction with the integrated tools of documentation • General appreciation for the consultation method • PROCESS REPORT • Introduce the use of flow charts for process description • Improve communication ability • QUALITY REPORT • Users uncomfortable with the Relevance concept • Need for incorporating some definitions available in the “Reading guide” into the report

17. Consulting the users on quality documentation supply • Future work • Analyse web survey data and conclude the analyses for the focus groups data • Identify the most important comments from the users and make them to converge to concrete improvement changes on Istat documentation tools • Provide feedback on metadata items and quality indicators defined at Eurostat level • Fine tune the tools • Launch the implementation strategy

18. Thank you