ESS.VIP Validation

1. ESS.VIP Validation Objectives, scope & concepts Angel Simón Delgado EUROSTAT Email: ESTAT-ESSVIP-VALIDATION@ec.europa.eu

2. ESS.VIP VALIDATION • VIP VALIDATION – First Phase • VIP VALIDATION – Deliverables in First Phase • ESS.VIP VALIDATON – Definition of the project • Validation Service (EDIT)

3. ESS VIP-V First Phase Overall goal: To develop validation solutions to be used by different production chains (horizontal integration), within the ESS (vertical integration) Bottom-up approach: • Extensive consultation of all possible stakeholders • Participative management • Business driven approach • From pilots experience to general principles

4. ESS VIP-V First Phase Scope, objectives and outputs • Documentation / Standardisation: • template and guidelines for process description • template and guidelines for a standard documentation of the validation process • Methodological analysis of Data Validation • Typology of validation rules • Standard definition of validation levels • Standard formalised “syntax” (understandable by business users) to express validation rules • Distribution of responsibilities in the production chain • Guidelines to be used for the attribution of responsibility in the whole production chain (MSs and Eurostat) by the WG. • Guidelines based on efficiency principles (Validation  Corrections: “the sooner, the better”) • Preparing for IS/IT solutions and architecture

5. ESS VIP-V First Phase Scope, objectives and outputs • Towards IT/IS solutions and architecture: • Users’ requirements to develop a new software to allow business users to input validation rules and the corresponding error messages in a shared Central Repository of Validation Rules. The new software should be able to generate the rules in the Validation syntax developed by the project. b) Validation Architecture defining the elements and their relationships in an integrated validation system ("common platform") to be used by: • Internal users, in an appropriate IS architecture to facilitate horizontal integration of IT/IS systems • All stakeholders in the production chain

6. Contribution to the VISION

7. ESS.VIP Validation First Phase - Deliverables Documentation Examples Methodology 1.1 Inventory of documents 1.2 Analysis of inventory 1.5 Inventory of validation rules 1.6 Inventory of error messages 1.3 Validation & statistical processes 2.4 Analysis of validation typologies 3.1 Validation rules by typology 3.3 Error messages 1.4 Validation typologies 2.4 Analysis of validation typologies 2.5 Levels of validation Solutions Templates & Guidelines 3.2 Validation syntax 4.1 Functional specifications for GUI 2.1 Documentation of validation process 2.2 Documentation of statistical process 3.3 Error messages 3.4 Selection of validation rules 3.5 Improvement actions 3.6 Attribution of responsibilities

8. Deliverables: Validation levels Same file Level 0: Format & file structure Level 1: Cells, records, file Same dataset From the same source Within a domain Between files Level 2: Revisions and Time series Within an organisation Validation complexity Data Between datasets Level 2: Between correlated datasets From different sources Level 3: Mirror checks Between domains Level 4: Consistency checks Between different organisations Level 5: Consistency checks

9. Deliverables: Typology of validation rules File Structure >1 file checks Filename Referential integrity File type Code list Delimiters Cardinality Format Mirror Time series 1 file checks Revised data integrity Model-based consistency Type 1n file checks Length Presence Allowed character Consistency Uniqueness Control Range Conditional

10. Guidelines for the allocation of responsibility, for the implementation of validation rules within the ESS based on an AGREEMENT Eurostat-NSI's with periodic performance revisions from both sides • Proposal for a generic business architecture of data validation: Deliverables: Guidelines for the attribution of responsibility of validation activities in the whole production chain Validation Controls – Different actors – Different responsibilities Step 1 Step 2 Step 3 Step 4 Data preparation by the NSI's Transmission of data and validation report Loading data to production database Additional processing & dissemination

11. Standard templates for error/warning messages and for validation report Deliverables: Standard template for error/warning messages • Validation report structure: • Header • Time stamp • User ID • Data checked • (dataset name…) • Body • Rules applied • Total failures • No. errors • No. warnings • Total records • Records failed • Sum of weights • Maximum admissible error weight • Rate of acceptance • Maximum possible amount of error • Rate of performance • Footer • Error/warning messages • Error/warning message structure: • Rule ID • Severity • Rule type ID • Message text • Action • Failing data

12. Deliverables: VALS - Validation syntax To define a meta-language for the domain of statistical data validation to express, document and communicate validation rules Standard syntax for validation language Trade-off between Human-understandable and Computer-parseable language Implementation through Graphic User Interface to support business users to input and maintain validation rules and rule-sets

13. Proposed approach • Goals • Implementation of the methodological developments of VIP-V Phase I in the statistical domains/WGs • Maintenance and refinement of standards developed • User requirements for further developments • Evaluation, monitoring and reporting

14. Proposed approach • Goals • Vertical integration of the micro data validation within the ESS production processes taking into account the results of the first phase of ESS VIP-V • Extension of the functional specifications to apply to micro data validation • Integrated solutions for micro data validation

15. Proposed approach • Goals • Adaptation of existing validation tools to the functional specifications issued from ESS VIP-V Phase I • Deployment of validation solutions to MSs • Distribution of validation rules in agreed language • Building-Blocks in an adequate web services architecture • Provision of web services validation solutions to be used by Member States before transmission to Eurostat

16. Proposed approach • Goals • Overall coordination of the project • Coherence of validation approaches within ESS • Implementation of the meta-language within ESS • Analysis of links with other VIP & ESS VIP projects • Good practices identification • More sophisticated validation solutions: • Longitudinal validation • Mirror checks validation • ESS wide shared final warehouses

17. Elements in a validation system: ESS production chain Eurostat Member States Statistical domain 1 Single Entry Point Statistical domain … Member States statistical production and validation Validation Services Web Inteface Statistical domain n • Eurostat

18. Validation Service Data Definition registry Validation rules repository • Validation Service Data Errors Validation report Metrics

19. VALIDATION RULESETS MAINTENANCE Validation architecture elements – Global overview Repository of rulesets and their metadata Data Structure SYNTAX ANALYSER SYNTAX specifications VALIDATION SERVICE SEP Single Entry Point Validation Report Data

20. Current actions and next steps • Assessment of Eurostat domains in the field of validation • Set of standard documentation for domain managers for harmonisation of communication with data providers • Task Force to: • Identify best practices in ESS • Advice on implementation in the ESS • Optimisation of the validation process

21. Current actions and next steps • Functional specifications for Validation services (EDIT) to be accordingly adapted to the findings of the project • Tools development: • System to create/maintain a central repository of validation rules • Development and/or adaptation of IT Tools (EDIT, eDAMIS) • Validation Quality Metrics

22. Thank you More information on: Email: ESTAT-ESSVIP-VALIDATION@ec.europe.eu Wiki (only from EUROSTAT): http://www.cc.cec/wikis/display/ESTATmethodology/ESS.VIP+VALIDATION)