Giovanna Brancato, Marina Signore Istat

1. Metadata and Quality Indicators Reuse for Quality reporting Work Session on Statistical Metadata (METIS) Giovanna Brancato, Marina Signore Istat Geneva, 10-12 March 2010

2. Work Session on Statistical Metadata (Metis) Outline • Istat information System on Survey Documentation (SIDI/SIQual) • Istat Quality Reporting for external users • Metadata and Quality documentation reuse for Istat quality reporting • Metadata and quality reporting reuse for ESMS Standards Geneva, 10-12 March 2010

3. SIQual navigation system (intranet/internet) Quality report Process report Statistical Yearbook Quality indicators Quality pilots Quality metadata Process metadata Programming Management Systems Quality documentation and dissemination at Istat: SIDI and SIQual Work Session on Statistical Metadata (Metis) SIDI Geneva, 10-12 March 2010

4. Work Session on Statistical Metadata (Metis) SIDImetadata • Key concept • Statistical production process: process aimed at producing statistical information • primary survey: data collection from reporting units (households, businesses, institutions) including administrative sources surveys 225 documented primary surveys 156 active, 53 ceased, 16 suspended • secondary study: processing of statistical data already available from previous primary surveys or secondary studies 101 documented secondary studies 81 active, 14 ceased, 6 suspended Geneva, 10-12 March 2010

5. Work Session on Statistical Metadata (Metis) SIQual SIDI navigation environment Intranet: metadata and standard quality indicators functionalities for queries on organisational units (direction/service/unit) Internet: metadata SIQual Geneva, 10-12 March 2010

6. SIDI Standard Quality Indicators Work Session on Statistical Metadata (Metis) Completeness status * Including Primary Surveys and Secondary Studies Geneva, 10-12 March 2010

7. Work Session on Statistical Metadata (Metis) Istat quality reporting for external users • Tools • Short methodological notes (SMNs) • Process reports (PR) • Quality reports (QR) Geneva, 10-12 March 2010

8. Work Session on Statistical Metadata (Metis) Istat short methodological note (SMN) Use Compact reports included in the Statistical Yearbook Structure General conceptual metadata: observed issues, analysis unit Methodological process: reporting unit, periodicity; EU regulation, sampling design, data collection modes Quality metadata: synthesis of the main activities carried out to prevent, monitor and evaluate nonsampling errors (unit nonresponse and measurement error), validation techniques Dissemination of provisional and final results: Timeliness, geographical and sector level of aggregation Languages Italian (English) Geneva, 10-12 March 2010

9. SIDI metadata reuse in the SMN Work Session on Statistical Metadata (Metis) Example SIDI documentation on observed issues: Attendance at museums and art exhibitions Chronic diseases Cinema attendance Friendship relations Games and other leisure activities Health conditions Housing conditions …    Geneva, 10-12 March 2010

10. SIDI metadata reuse in the SMN Work Session on Statistical Metadata (Metis) • Example • SIDI documentation on the activities to prevent unit non response: • - Survey presentation letter signed by Istat President • - Guarantees on statistical confidentiality • - Interviewer identification badge      • - Description of survey objectives by interviewers • - Telephone contacts to make an appointment for the interview • Establishing a toll free line or telephone number for further explanations Compacted description in the short methodological note: “Activities for preventing unit nonresponse” Geneva, 10-12 March 2010

11. single value average over a period SIDI quality indicators reuse in SMN Work Session on Statistical Metadata (Metis) • Examples • SIDI documentation on Timeliness: Geneva, 10-12 March 2010

12. Work Session on Statistical Metadata (Metis) Istat process report (PR) Use Standard process documentation to be included in publications on survey results as methodological annex. General structure - Introduction - Objectives - Survey design      - Data capturing - Data pre-treatment - Processing - Documentation Geneva, 10-12 March 2010

13. An editor window allows to adjust and modify the text to turn it in a fluent language Work Session on Statistical Metadata (Metis) SIDI metadata reuse in PR The report is pre-filled in with the information drawn from SIDI Geneva, 10-12 March 2010

14. Work Session on Statistical Metadata (Metis) Istat quality report (QR) Use It is a short report, including a subset of quality indicators, to be disseminated together with press releases on Istat web site Structure It is organised into the following sections: - Relevance - Accuracy - Timeliness and punctuality      - Clarity and accessibility - Coherence and comparability Geneva, 10-12 March 2010

15. Work Session on Statistical Metadata (Metis) Indicators included in the quality report (QR) • Qualtiy indicators • - Sampling error (confidence interval or coefficient of variation) • - Weighted/Unweighted Response rate [respondent units / (eligible units and unresolved units)] • - Net imputation Rate[imputations from blank to nonblank values/imputable values] • - Modification Rate  [changes from nonblank to nonblank values/imputable values] • - Mean Revision (size and direction of the revisions) • Absolute mean revision (average size of revisions) • Timeliness (of provisional/final results, of data sources) • - Comparability (length of the series) • - Coherence between provisional and final estimates (difference or relative difference btw provisional and final estimates) Geneva, 10-12 March 2010

16. SIDI metadata and quality indicators reuse in QR Work Session on Statistical Metadata (Metis) • Implementation phase • Development of the product quality report integrated to SIDI management environment • Limited reuse of metadata • Reuse of standard quality indicators following the short methodological approach • Need of developing new quality indicators and revising some existing ones Geneva, 10-12 March 2010

17. Work Session on Statistical Metadata (Metis) ESMS requirements Definition Statistical metadata to be used by Member States for documenting statistical data to Eurostat  metadata  quality indicators (ongoing work)  harmonisation within Eurostat, OECD, IMF Uses  Comparisons and evaluations  Metadata and quality information for the users Geneva, 10-12 March 2010

18. Work Session on Statistical Metadata (Metis) Nature of ESMS metadata Organization: Institutional mandate, confidentiality legislation, release policy, quality management Data set and variables: classification system, sector coverage, statistical concepts and definitions, statistical unit and population, reference area, time coverage, base period, unit of measure, reference period Dissemination frequency and formats Eurostat quality components: relevance, accuracy, … Statistical processing Standard quality indicators [Simulation: compilation of ESMS template with metadata and quality indicators for a survey ~ 60%] Geneva, 10-12 March 2010

19. Work Session on Statistical Metadata (Metis) Issues on ESMS Type of data and frequency of data collection Organisational data: univocally defined throughout the institution Metadata: stable over time Quality indicators: survey edition Standard quality indicators Harmonisation: among different Member States, among different EU regulations  Eurostat Handbook/Standards for Quality Reports (2009) Priority setting: identification of a minimum set Use: additional indicators upon request for expert users Aggregation: survey edition, annual averages (for monthly surveys) Dissemination: summary measures, ranges of variations, graphical representation Geneva, 10-12 March 2010

20. Metadata and quality indicators reuse for quality reporting Thank you for your attention ! Geneva, 10-12 March 2010

21. Work Session on Statistical Metadata (Metis) SIDImain features Metadata documentation standards Thesaura: lists of standard items to be used to document process activities and quality control actions Content: Topics of the survey, analysis units, questionnaire Process: Reporting unit (sources of the secondary study), survey design, data collection, data transformation, data processing Quality: Activities carried out to prevent, monitor and evaluate survey errors Metadata qualitative descriptions: free notes supporting standard metadata items Geneva, 10-12 March 2010

22. Work Session on Statistical Metadata (Metis) Istat process report (PR) Use Standard process documentation to be included in publications on survey results as methodological annex. General structure - Introduction(name, period, description) - Objectives (information needs and periodicity, issues) - Survey design (population, sampling, reporting unit and frame, questionnaire design and testing, pilot survey) - Data capturing (data collection mode, field operations, sensitive questions, unit nonresponse monitoring, mandatory participation, response error) - Data pre-treatment (coding, manual revision, data entry, editing and imputation) - Processing (estimation, validation, use of generalised software) - Documentation (list of relevant documents) Geneva, 10-12 March 2010

23. Work Session on Statistical Metadata (Metis) Istat quality report (QR) Use It is a short report, including a subset of quality indicators, to be disseminated together with press releases on Istat web site Structure It is organised into the following sections: - Relevance(regulations and use of the data) - Accuracy (methodological context, sampling error, nonsampling error, revision policy) - Timeliness and punctuality (production of preliminary results, timeliness) - Clarity and accessibility (links to data and documentation) - Coherence and comparability (national or international regulations and manuals, over time data comparability) Geneva, 10-12 March 2010