ESCWA SDMX Workshop

1. ESCWA SDMX Workshop Session: SDMX Introduction

2. What is SDMX? The problem space (2001): Statistical collection, processing, and exchange is time-consuming and resource-intensive Various international and national organisations have individual approaches for their constituencies Uncertainties about how to proceed with new technologies (XML, web services …)

3. What is SDMX? The Statistical Data and Metadata Exchange (SDMX) initiative addresses these challenges and opportunities that have just been mentioned: By focusing on business practices in the field of statistical information By identifying more efficient processes for exchange and sharing of data and metadata using modern technology

4. Who is SDMX? SDMX is an initiative made up of seven international organizations: Bank for International Settlements European Central Bank Eurostat International Monetary Fund Organisation for Economic Cooperation and Development United Nations World Bank The initiative was launched in 2001

5. Official Recommendations SDMX has been officially recommended: February 2007: SDMX endorsed by the European Union’s Statistical Programme Committee (now ESS Committee) March 2008: UN Statistical Commission declares SDMX to be the preferred standard for data and metadata

6. For Data Consumers Imagine being able to find, query, and view data direct from the databases of the data producers using a single software tool. No more the need to access each website independently, use their navigation and query mechanism, and the viewing options provided. Use a single tool that assists you to find the data, then allows you to query the database of the producer, and then to view the data as graphs, charts and tables (often some of these searching and viewing options are not available on the producer's website). Too good to be true? Not so with SDMX. Many organisations such as the IMF, ECB, OECD have launched web services which can be queried using SDMX standards. For a demonstration of this click here <<link to Fusion>> For Data Collecters Imagine being able to eliminate the many and varied formats in which data and reference data are reported. Imagine being able to immediately load the reported data into a database so that  it can be viewed in a meaningful way to assist the validation and cleaning stage. Imagine sharing the data and reference metadata collection with your statistical partners so that they collect from their community and you collect from yours, and then you share the data. This can all be achieved by describing the data and reference metadata structures in SDMX and using the SDMX data and reference metadata formats for reporting and exchange. For Data Reporters Imagine being able to report the same data in the same format to all of the organisations that require your data. Even better, that you do not send the data at all. After all, the World Wide Web works on the principle of publishing and discovery. Interestingly, SDMX also supports this paradigm: make the data available either as an SDMX file available at a URL or as web service that can be queried, publish this fact in an SDMX Registry, and then interested parties can discover the data, or be informed automatically by the SDMX  Subscription/Notification service. For Data Publishers Imagine being able to build a web dissemination service that can support any type of data and refernce metadata, and bring these together in a dissemination environment. Imagine being able to provide this without needing to change any software when new data are added. Imagine being able automatically to create and load the database that is used for the web dissemination service. Imagine being able to enable this database to be queried directly from applications outside of the organisation so that data is available to all. All this is possible and extremely practical using SDMX. In fact, web dissemination is an ever-increasingly popular use of the SDMX standards because with the SDMX structural metadata such as a Data Structure Definition, SDMX Web Service standards, and SDMX standard data and reference metadata formats, such systems are easy to build and have very low minatenance effort. SDMX: Benefits • For Data Consumers • Search for, locate, and query data and related metadata • Query directly the database and metadata repositories of the data and metadata producers • View data and related metadata in table, charts, graphs, maps using SDMX visual components • For Data Collectors • Share the data collection burden • Eliminate many and varied formats for data and reference metadata • Automate the database and metadata repository loading process to assist validation

7. For Data Consumers Imagine being able to find, query, and view data direct from the databases of the data producers using a single software tool. No more the need to access each website independently, use their navigation and query mechanism, and the viewing options provided. Use a single tool that assists you to find the data, then allows you to query the database of the producer, and then to view the data as graphs, charts and tables (often some of these searching and viewing options are not available on the producer's website). Too good to be true? Not so with SDMX. Many organisations such as the IMF, ECB, OECD have launched web services which can be queried using SDMX standards. For a demonstration of this click here <<link to Fusion>> For Data Collecters Imagine being able to eliminate the many and varied formats in which data and reference data are reported. Imagine being able to immediately load the reported data into a database so that  it can be viewed in a meaningful way to assist the validation and cleaning stage. Imagine sharing the data and reference metadata collection with your statistical partners so that they collect from their community and you collect from yours, and then you share the data. This can all be achieved by describing the data and reference metadata structures in SDMX and using the SDMX data and reference metadata formats for reporting and exchange. For Data Reporters Imagine being able to report the same data in the same format to all of the organisations that require your data. Even better, that you do not send the data at all. After all, the World Wide Web works on the principle of publishing and discovery. Interestingly, SDMX also supports this paradigm: make the data available either as an SDMX file available at a URL or as web service that can be queried, publish this fact in an SDMX Registry, and then interested parties can discover the data, or be informed automatically by the SDMX  Subscription/Notification service. For Data Publishers Imagine being able to build a web dissemination service that can support any type of data and refernce metadata, and bring these together in a dissemination environment. Imagine being able to provide this without needing to change any software when new data are added. Imagine being able automatically to create and load the database that is used for the web dissemination service. Imagine being able to enable this database to be queried directly from applications outside of the organisation so that data is available to all. All this is possible and extremely practical using SDMX. In fact, web dissemination is an ever-increasingly popular use of the SDMX standards because with the SDMX structural metadata such as a Data Structure Definition, SDMX Web Service standards, and SDMX standard data and reference metadata formats, such systems are easy to build and have very low minatenance effort. SDMX: Benefits • For Data Reporters • Common reporting format (including coding schemes) for all reporting needs • Ability to open databases and metadata repositories to direct query for data collectors • For Data Publishers • Automated web dissemination system for data and reference metadata • As soon as data is published it is available for query, and linked to the reference metadata

8. Exchange Patterns Bilateral: Institutions exchange data according to bilateral agreements regarding format, timing, protocols, etc. Gateway: Institutions share the data they collect with their peers, in agreed formats among counterparty communities Data-sharing: standard exchange of data using standard formats and protocols

9. Bilateral Exchange

10. Gateway Exchange

11. Data/Metadata-Sharing Exchange query/retrieval RSS or SDMX Notification Registration of Data Source Data/Metadata Reporters Data/Metadata Consumers

12. Adopters/Interest The following are known adopters (or planning to adopt): US Federal Reserve Board and Bank of New York European Central Bank Joint External Debt Hub (WB, IMF, OECD, BIS) UN/TRADECOM at UN Statistical Division NAAWE (National Accounts from OECD/Eurostat) European Statistical System (Eurostat and National Statistical Institutes) Mexican Federal System IMF (BOP, SNA, SDDS/GDDS) Food and Agriculture Organization Millennium Development Goals (UN System, others) International Labor Organization Bank for International Settlements OECD World Bank Marchioness Islands (Spanish/Portuguese Statistical Region) UNESCO (Education) Australian Bureau of Statistics International Energy Agency World Health Organization There are many others

13. SDMX Status and Future Revised version 2.1 standards have been released (end April) Increasing availability of SDMX tools In May 2011 the IMF and World Bank hosted a 3-day SDMX Global Conference 250 people attended Many implementation scenarios Focus on web services There was a vendor demonstration area on day 3 Commitment of the Sponsors 2011 – Web “Sandbox” for users to play with SDMX 2012 - Global SDMX Registry Retrieval of common structural metadata by local registries Sharing of structural metadata Portal to find data and SDMX web services 2015 - 20 International Data Structure Definitions World Bank and IMF asked to help with training and tools for developing nations

14. SDMX and Domain Standards SDMX are keen that organisations prominent in a specific domain take the lead to define the domain standards Structural definitions Code lists Concepts Data structure definitions Metadata structure definitions Quality metadata and quality frameworks Based on the SDMX content oriented guidelines

15. SDMX Version 2.0/2.1: High Level Overview Dataor Metadata Structure Definition Process Structure and Item Scheme Maps Category Scheme Categorisation Data Set or Metadata Set Data or Metadata Flow Category Attachment Constraint Content Constraint Provision Agreement Data Provider Registered Data Source or Metadata Source

16. Dataor Metadata Structure Definition High Level Overview: Flow Category Scheme Data or Metadata Flow Category Categorisation Data Provider Scheme  Drill Down Query Provision Agreement Data Provider  Discover  Register Registered Data Source or Metadata Source Data or Metadata Source references

17. Existing Websites Using SDMX BIS http://stats.bis.org ECB http://www.ecb.int/stats/exchange/eurofxref/html/index.en.html http://stats.ecb.europa.eu/stats/sdmx/visualisation/icp/dashboard/rc1/ Code available at (http://flex-cb.googlecode.com/) OECD http://stats.oecd.org/WBOS/index.aspx http://stats.oecd.org/OECDregionalstatistics/

18. SDMX: Course Introduction At the end of the Sessions on SDMX you will have Understood the scope of SDMX Gained a high level understanding of the SDMX Information Model Know the key SDMX structures that can be used to create, report, store, maintain, visualise, MDG data and structural metadata

19. SDMX: Course Introduction In order to do this you will learn about SDMX Structural Metadata DSD and related Concepts and Code lists MSD and related Concepts and Code lists Category Scheme and Classification Data flow and Metadata flow Provision Agreement Registration Subscription and Notification Registry Database Query SDMX Data and Metadata Data set formats Metadata set SDMX Web Services

20. Tools Used on the Course SDMX Registry/Repository

21. Tools Demonstrated SDMX Data Viewer SDMX Structure and Data Converter SDMX Relational Database

22. Now – The Hard Work