Proposal of a revised approach for data validation within the European Statistical System

1. Proposal of a revised approach for data validation within the European Statistical System Michel HENRARD European Commission – EUROSTAT

2. The current situation • Validation process : 2 separated steps • Member States • Eurostat • Double work ? Zero Work ? Ping-Pong ? Statistical data editing - UNECE work session – OSLO 24-26 September 2012

3. The current situation Harmonisation Validation procedures Harmonised? Documented ? Formally agreed? Coordinated ? Validation rules Common syntax ? Formalised ? Statistical data editing - UNECE work session – OSLO 24-26 September 2012

4. The current situation Harmonisation of software solutions Standard building blocks within Eurostat Partial degree on integration Many ad-hoc solutions High number of supported solutions Does not help streamlining of production processes High cost of development and maintenance Common validation solutions within the ESS ? Statistical data editing - UNECE work session – OSLO 24-26 September 2012

5. The current situation Compliance monitoring Assessment of data quality Subjective interpretation Not structured Statistical data editing - UNECE work session – OSLO 24-26 September 2012

6. Yielding efficiency gains Vision Infrastructure Project on Validation (end 2010) Validation solutions that can be used both by the Member States & Eurostat Validation as soon as possible Definition of the roles in the validation chain Statistical data editing - UNECE work session – OSLO 24-26 September 2012

7. Project proposals Better & standard documentation of the validation process Should be part of the general documentation of the complete production system Statistical data editing - UNECE work session – OSLO 24-26 September 2012

8. Project proposals Agreed & standard description of checking rules Common language (syntax) Guidelines for the selection of the validation rules Statistical data editing - UNECE work session – OSLO 24-26 September 2012

9. Project proposals Availability of common software solutions Common standard IT tools and associated services Automation of procedures within the ESS Statistical data editing - UNECE work session – OSLO 24-26 September 2012

10. Project proposals Clear & agreed distribution of responsibilities Validation where it can be done in the most efficient way Statistical data editing - UNECE work session – OSLO 24-26 September 2012

11. How ? Process Description Systematic standard description of The data validation The validation rules using a common syntax Statistical data editing - UNECE work session – OSLO 24-26 September 2012

12. How ? Validation review with Member States Gradual introduction of objective measurement of data quality Clear responsibility for each validation rule based on efficiency considerations (the sooner the better) Calendar of introduction of the new mechanism Statistical data editing - UNECE work session – OSLO 24-26 September 2012

13. How ? Integration of validation in compliance assessment procedures Based on the agreed set of validation rules Reports on validation problems in the statistical domains Statistical data editing - UNECE work session – OSLO 24-26 September 2012

14. How ? Improve the coherence of validation solutions within Eurostat Taxonomy of validation rules Standard description of the validation rules (common language/syntax) Good practices Current validation building blocks adapted to the users’ requirements Statistical data editing - UNECE work session – OSLO 24-26 September 2012

15. How ? Provision of standard solutions to be shared in the ESS Deployment of a standardized validation language Adapting the IT tools to be shared in Eurostat & to be offered to Member States Statistical data editing - UNECE work session – OSLO 24-26 September 2012

16. What the VIP on Validation does provide ? Templates & Guidelines Documentation of the validation process Error messages and reporting Selection criteria of validation rules ensuring minimum quality standard Attribution of responsibility of validation tasks Statistical data editing - UNECE work session – OSLO 24-26 September 2012

17. What the VIP on Validation does provide ? Inventory of validation rules used in Eurostat Typology of validation rules Human understandable language for validation rules User requirements & functional specifications to adapt the IT tools to the needs identified Statistical data editing - UNECE work session – OSLO 24-26 September 2012

18. Q&A Thank you for your attention Statistical data editing - UNECE work session – OSLO 24-26 September 2012