1 / 18

SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans

SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans Bengt-Åke Lindblad, Adam Wroński Eurostat. Eurostat Unit B3 – IT and standards for data and metadata exchange. Content. What is SDMX-RI Why use SDMX-RI Usage status Development strategy Release calendar.

pillan
Download Presentation

SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans Bengt-Åke Lindblad, Adam Wroński Eurostat Eurostat Unit B3 – IT and standards for data and metadata exchange

  2. Content • What is SDMX-RI • Why use SDMX-RI • Usage status • Development strategy • Release calendar Eurostat Unit B3 – IT and standards for data and metadata exchange

  3. What is SDMX-RI • Universal framework for modern data provision • Set of pick-and-choose reusable building blocks allowing a statistical office to expose data to the external world based on access rights • Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse • Uses SDMX standards incl. one for Web Services Eurostat Unit B3 – IT and standards for data and metadata exchange

  4. Eurostat Unit B3 – IT and standards for data and metadata exchange

  5. Why SDMX-RI • Developed to simplify the exchange of data • Provides standard software and components, allowing individual statistical organisations to interact and exchange their data using the same software and methodology • Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes • Developed in both Java and .NET Eurostat Unit B3 – IT and standards for data and metadata exchange

  6. Usage • As of Sept 2012 deployed in 22 EU countries • Tested across EU Member States (2011 population census) • Running in Mexico • Expression of interest • Latin America, the Caribbean, OECD and Russia • July 2013: • expected to run in EU27 member states • in Eurostat dissemination Web Service upgrade Eurostat Unit B3 – IT and standards for data and metadata exchange

  7. SDMX-RI versions SDMX-RI Ultimate Solution SDMX-RI Intermediate Solution • Common API SDMX v2.1 SDMX-RI Initial Solution • Streaming Eurostat Unit B3 – IT and standards for data and metadata exchange

  8. Development strategy 2012-2013 Architectural changes are needed (two step approach) Intermediate solution to solve performance & out of memory errors Ultimate solution to provide a common API and implement SDMX 2.1 Implement new user requests and correct defects Widen the scope and usage of SDMX-RI among data providers ESSVIP project such as Web INfrastructure for Statistics Reuse for other statistical data collections DSWS (Eurostat dissemination web service) Eurostat Unit B3 – IT and standards for data and metadata exchange

  9. Why an intermediate solution Eurostat Unit B3 – IT and standards for data and metadata exchange • Problems needed to be solved within short time for existing users • Performance • High memory consumption and response times constrained organizations from putting it in production • Out Of Memory errors • Impossible to serve large data requests.

  10. What changed? Eurostat Unit B3 – IT and standards for data and metadata exchange • Streaming of data in the service • Migration to JAX-WS in Java • IO revised with Streaming Writers • Data Retriever API changed (streaming)

  11. Benefits Eurostat Unit B3 – IT and standards for data and metadata exchange • Better performance • Improvement up to 4 times faster for .Net • Improvement up to 7 times faster for Java • Solution to Out Of Memory problems for large datasets • No memory constrains • Data are streamed to the client

  12. Impact Eurostat Unit B3 – IT and standards for data and metadata exchange • Organizations that have installed initial solution • Only re-install the Web Service Package • Existing users of Web Service package not affected • Same v2.0 interface remain • Organizations using the components API • API has changed due to streaming support • Migration required • Organizations has done modifications to the source code • Have to make the changes again if they want to use the intermediate version.

  13. Why an ultimate solution? Eurostat Unit B3 – IT and standards for data and metadata exchange • Use of common API (that supports various versions and flavors of SDMX) • Interfaces which allow implementation using the components and also the interfaces for component-to-component communication • Better inter-organization component reusability • Ability to support new versions of SDMX without changing our applications • Support of SDMX 2.1 • New messages (data representation, queries) • New Interfaces (SOAP/REST)

  14. What will be changed? Eurostat Unit B3 – IT and standards for data and metadata exchange • All modules migrated to the common API • To use of new SDMX Model/IO • Components APIs will be changed • Support of SDMX 2.1 messages and new query features • Extend to support 2.1 standardized SOAP and RESTful APIs • New Web Service end points • Co-existence with v2.0 service • Support of v2.1 error codes

  15. Impact Eurostat Unit B3 – IT and standards for data and metadata exchange • Mapping Assistant users will be able to reuse their Mapping Stores • Mapping Store upgrade will be supported • Will be required to re-install MA only • Organizations that have installed initial solution • Only to re-install the Web Service Package • Existing users of Web Service package will not be affected • Same v2.0 interface will remain • Organizations using components API • API has changed due to streaming support • Migration will be required • Migration guidelines will be provided • Organizations done modifications to the source code • Will have to make the changes again if they want to use the new version.

  16. Status - release Eurostat Unit B3 – IT and standards for data and metadata exchange • Common API development (jar/dllexpected end 2012) • Java: ongoing • .NET: not started yet (API should be ported to C# and then implemented) • SDMX 2.1 support (expected Q1 2013) • Not yet started (dependent on API) • Enhancements and bug corrections (independent) • Mapping Assistant (Sept 2012) • Improved transcoding, error handling, QueryEditor, etc • Mapping Assistant (end 2012) • Implementation of 11 request mainly linked to the user interface and the usage • Test Client • using encrypted user id and password

  17. SDMX-RI contacts Bengt-Ake.Lindblad@ec.europa.eu Adam.Wronski@ec.europa.eu Eurostat Unit B3 – IT and standards for data and metadata exchange

  18. Web Service Package extension to SDMX 2.1 and new interfaces SOAP Request v2.0 SOAP Request v2.1 REST Request v2.1 NSI_Service_2.0 NSI_Service_2.1 NsiRestService Web Service Provider Controller Structure Retriever Data Retriever (streaming) Common API API Implementation Eurostat Unit B3 – IT and standards for data and metadata exchange

More Related