The implementation of SDMX in the statistical practice

1. Statistics: Investment in the Future – Prague, 14-15 September 2009 SDMX implementation in the statistical practiceMarco PellegrinoEurostat, Statistical Information Technologies Unit (B5) marco.pellegrino@ec.europa.eu

2. The implementation of SDMX in the statistical practice Background SDMX content-oriented guidelines (2009) The implementation of COG (cross-domain concepts, domains, metadata common vocabulary) Advantages for metadata exchange IT infrastructure Portable software package Tools for reference metadata Use of the SDMX registry Re-usable software components Training and other capacity-building actions

3. SDMX deals with… Exchange and Sharing of statistical information Structural metadata Reference metadata Describe data structure Describe the contents and the quality of the statistical data Statistical data Statistical metadata Emphasis on macro-data (aggregated statistics) Promotes a “data sharing” model • low-cost • high-quality of transmitted data • interoperability between (otherwise) incompatible systems

4. Benefits from SDMX standards (1/2) • improve quality and efficiencies in the exchange and dissemination of data and metadata • harmonisation and coherence of data • Focus on meaning • open format (XML) rather than proprietary ones • reduce national reporting burden to European and international institutions

5. Benefits from SDMX standards (2/2) • can be used by national and international agencies as key building blocks in internal statistical IT systems used for collecting, compiling, storage and searching of statistical information • can potentially reduce the cost of developing statistical software systems • avoid duplication of efforts in developing and maintaining standards and tools for processing statistical information • are anchored in international recognised bodies such as ISO • full compatibility with standards previously or currently used

6. Eurostat Strategy Eurostat SDMX implementation strategy involves many different actions and projects: • Development of SDMX tools • Use of SDMX for the evolution of the information system • Deployment of theSDMX Registry • SDMX architecture at Eurostat for exchanging data and reference metadata (SEP) • Evolution of the information system (CVD strategy) • Dissemination of SDMX files • Development of the Euro-SDMX Metadata Structure (ESMS) • Domain-specific implementation projects • Capacity-building actions

7. Euro-SDMX Metadata Structure

8. The Eurostat CVD architecture To Data files Metadata Handler (MH) Domain Z production system Domain Y production system Reference Environment databases Internet Portal CVD compliant production system in domain X (GSAST, NAPS or Comext) New User Interface Comext dissemination Single Entry Point eDAMIS Data ready for dissemination Comext int. trade and industrial stat. reference Validated data Data Explorer Collected data Tables-Graphs-Maps Production system standard modules, e.g. NewCronos (Eurobase) reference CVD standard modules e.g. Country/ Regional Profiles Building Block A Building Block B Loader WMS Process control Domain specific software The CVD-MH within the Eurostat IT architecture

9. https://webgate.ec.europa.eu/sdmxregistry/index.do

10. Domain-specific SDMX implementations in ESS The following development activities concern the implementation of SDMX in specific statistical domains:

11. Capacity-building actions • Training • Courses for statisticians and IT staff • Tutorial for Working Groups • SDMX self-learning package (in preparation) • International cooperation actions, workshops • ESSnet on SDMX (starting in October)

12. The NSI perspective on SDMX benefits Reduce reporting burden to national, European and international institutions Can improve harmonisation, standardisation and integration processes inside a NSI Be part of an international “community” where NSIs can share experiences and software Open Source culture: tools are publicly available Eurostat, upon request, provides technical advice to NSIs interested in starting SDMX projects Eurostat designed a SDMX reference architecture for NSIs and developing building blocks through its implementations

13. Data Repository (Warehousing) Architecture register SDMX Registry query NSI P U L L Received data in SDMX-ML Eurostat Pull Requestor Loader Eurobase Dissemination Verification / Conversion To SDMX eDAMIS P U S H XSL for SDMX-ML Warehouse storage Intermediate storage Data Input

14. Data Hub Architecture RSS / data registration SDMX Registry NSI Data query Response NSI Data Portal Query NSI Retrieve dataset Dissemination cache NSI XSL for SDMX-ML

15. The Census Hub architecture based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes formats and technologies The hub is based on agreed hypercubes, but here the hypercubes are not sent to the central system, but data are fetched directly from the data producer databases when a user requests them SDMX formats and architecture are used

16. National Statistical Institute National Statistical Institute The Census Hub Project Eurostat Census Hub

18. The Mapping Process Data sets within data producers’ Information System are described using “local” structural metadata (concepts, code lists, formats) SDMX standards harmonize structural metadata within a statistical community, and describe data sets by DSDs (concepts, code lists, dimensions, attributes, measures, etc.) SDMX-ML structure files “local” structural metadata and SDMX structure metadata must be mapped*: concepts mapping codes mapping (*) see SDMX User Guide, section B.7.6, page 73

19. The NSI perspective on SDMX: where to start from Decide to start in using SDMX autonomously Design and build “unilaterally” DSDs and/or reuse those already available at European and International level Decide which part of the Information System will be affected (collection, processing, analysis, dissemination) and which kind of SDMX architecture would be more suitable Join SDMX projects launched by International organizations Several pilot projects launched by Eurostat within the ESS (Census Hub, EuroGroup Register, etc.) DSDs defined centrally by Eurostat after agreements taken within WG and TF The SDMX architecture implemented in the NSI must be compatible with the reference architecture of the whole project

20. Free/open SDMX software and tools (1) Data/Metadata Structure Definition and transformations SDMX Converter (Eurostat) Data Structure Wizard (Eurostat) SDMX Transformation Package, SDMX Authoring Tool , Data Structure Definition Tool , Metadata Structure Definition Editor (Metadata Technology) Implementation of SDMX registry specifications SDMX Registry (Eurostat) (Metadata Technology) (UNSD) KeyMaster (Metadata Technology) Data provisioning, Data Set registration (Metadata Technology) SDMX Query Tool, Query Client (Metadata Technology)

21. Free/open SDMX software and tools (2) Presentation of SDMX-ML data files to users Business Cycle clock (Eurostat) SDMX Visualization Tools (Eurostat) Visual framework (ECB) Frameworks and toolkits for working with SDMX SDMX framework (Istat/Eurostat) The NSI Web Service Prototype (Eurostat) Data Retriever Building Block (Eurostat) Mapping tools SDMX Mapping assistant (Eurostat)

22. Summarising • SDMX is global • Good progress reached in creating the SDMX standards • More emphasis now on implementation in statistical domains across statistical organisations • SDMX at the core of the harmonisation of the statistical business process. For more information: http://www.sdmx.org

23. Questions?Marco Pellegrinomarco.pellegrino@ec.europa.eu 23