1 / 13

Eurostat SDMX Toolkit SDMX-RI and collaborative development Alvaro Diez SOTO Eurostat, A3

Eurostat SDMX Toolkit SDMX-RI and collaborative development Alvaro Diez SOTO Eurostat, A3. Eurostat tools What, why and how with the SDMX-RI W hat is new C ollaboration and future plans. Presentation overview. Eurostat SDMX TOOLS. OSS, supported by Eurostat

dventura
Download Presentation

Eurostat SDMX Toolkit SDMX-RI and collaborative development Alvaro Diez SOTO Eurostat, A3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Eurostat SDMX ToolkitSDMX-RI and collaborative developmentAlvaro Diez SOTOEurostat, A3

  2. Eurostat tools What, why and how with the SDMX-RI What is new Collaboration and future plans Presentation overview

  3. Eurostat SDMX TOOLS • OSS, supported by Eurostat • For data collectors and data providers • Cover different use case • Based on Common API (CAPI) • Based on SDMX Standard: Registry, WS, Reference Architecture, IM • Java mainly (SDMX-RI also in .NET) • GUI, API and/or WS Registry SDM Web Client DSW Mapping Assistant NSI WS EDIT STRUVAL SdmxSource DSWS • But "one size does not fit all" • That is why the SDMX-RI is • Extensible • Re-usable and shared building blocks • Pluggable DLBB MA WS API ESS-MH Census Hub Data Hub Converter

  4. Data providers and consumers Universal framework for data dissemination and reporting Exposes data stored in non SDMX compliant database Open source, free to use, developed by a community Plugin architecture and implementation Exposes APIs for 3rd party development Who, why and what with the SDMX - RI

  5. SDMX RI architecture Data Consumers National Statistics Institute/Data Providers Services pulling data, users browsing & downloading data SDMX RI Mapping Assistant Configuration Map DB to DSD SdmxSource Plugins SDMX Data Web Service Plugins SdmxSource Browse Data Web Client Non SDMX Data Dissemination database SdmxSource

  6. Common API for SDMX development • Abstracts most of the SDMX complexity from the developers • Available for Java and .NET • Java 7 • Maven packages • .NET Standard 2.0 • NuGet packages • Eurostat actively maintains fork for Java and .NET port • Initial implementation in Java (Metadata Technology) • Many contributions from OECD and ISTAT (SDMX-JSON, SDMX-CSV, RDF, etc.) SdmxSource

  7. SdmxSource Implementations in SdmxSource Example Implementations in other tools Custom Excel format SDMX 2.0 Generic Data SDMX 2.0 Compact Data Custom CSV formats One streaming API for reading data Software uses SDMX 2.0 Cross Sectional Data SDMX 2.1 Generic Data SDMX 2.1 Structure Specific Data SDMX CSV Data SDMX JSON Data SDMX EDI Data

  8. SDMX RI Web Service • Available for • Java 7 Tomcat, WebLogic • ASP.NET Core 2.2, IIS on Windows, Kestrel on Windows/Linux • SDMX Compliant • Plus experimental features • Implemented and actively maintained by Eurostat • .NET many contributions from OECD and ISTAT • Performance • Streams data for data requests

  9. SDMX RI Web Service Architecture Authentication Policy authorization SDMX REST SOAP SDMX v2.1 SOAP SDMX v2.0 Web Service Other plugins Plugins Output Formats Authorization Data / Structure Store MSDB Dataflow Authorization SDMX XML, CSV, JSON formats, GESMES/TS SDMX RI Data/Structure Retriever and Persistence for Mapping Assistant Database. SDMX Header, Dissemination DB connection RDF from ISTAT SdmxSource

  10. Mapping Assistant Web application • Frontend in React/Javascript (MAWEB) • Backends in Java 7 and ASP.NET Core 2.2 (MAWS) • No installation on users PC • Backend exposes REST API for 3rd party development • JSON format used for backend/frontend communication MSDB Mapping Assistant Users Mapping Assistant Frontend (React) J S ON REST API Mapping Assistant Backend (.NET Core/Java 7) SDMX Web Service Dissemination database • Legacy desktop version • .NET Framework 4.6.2 only • Requires installation on each users PC • No new features added

  11. Mapping Assistant – How it works Mapping Assistant • Stores the SDMX structures agreed for the data exchange process • Allows users to define subsets of data to be disseminated • Creates and stores mappings between the internal data structure and SDMX concepts (e.g. My_column_A = AGE) • Creates and stores mappings between the internal classifications and SDMX codelists(e.g. My_code_AB = Y_LT15) • Uses Mapping Assistant web service for all communication MSDB Mapping Assistant Frontend (React) J S ON SDMX – RI Mapping Assistant Web service functionality • No graphical user interface • Invisible for the user modules controlling the incoming data requests • Exposes a REST API for manipulating the Mappings, subset of dataset • JSON format • Includes part of SDMX WS • Result: • Control the exposed data • Preview the data in SDMX format • Identify errors Mapping Assistant Backend (.NET Core/Java 7) SDMX Web Service DDB

  12. Collaboration and Future Contributions from OECD and ISTAT • Proposing and/or implementing new features • Identifying and/or fixing bugs • Design/Architecture discussions • New ideas Focus on Mapping Assistant Web application • Continue improving user experience • Automate mapping procedure • Support new use cases • Create a SDMX DSD from dissemination data • Design a DSD • Integrate ISTAT data and metadata manager • New data explorer • Continue to improve SDMX coverage

  13. THANK YOU!

More Related