Improving Data Collection at IMF: Standards, Technology, Collaboration

1. Data Collection at IMF Standards and Technology to Improve Timeliness and Reduce Reporting Burden Gangti Zhu Andreas Hake El Bachir Boukherouaa Addis Ababa October 2017

2. Outline • Context of data exchange • Data exchange Key Principals • Role of transparency and data availability • The current IMF data exchange infrastructure • The current challenges • A way forward for more collaboration & Partnership • Technology trends • Conclusion

3. Context of data exchange • Large volumes of economic data • Different frequencies • Variety of Formats • Excel templates • OpenData • APIs • Confidential data • Reporters are becoming more technologically advanced and make their data available for disseminations • Global sources of data

4. Data Exchange key principals S Standards SDMX, DSD T Technology Machine-to-Machine, APIs E Engagement Support, mutual interest, collaboration P Partnership O Outcome Multilateral partnership, peer-to-peer Reduce burden, better DM, aligned vision

5. Why SDMX Matters SDMX standards make it possible for efficient and automated data exchange • SDMX provides standards relating to statistical data and metadata exchange • SDMX is model-driven through the SDMX Information Model • Improved interoperability among the different implementations – thanks to “Guidelines for the Use Web Services” • Machine readable - allow easy Integration into applications

6. Approach for leveraging Standards, Technology and Partnerships Standard Platforms and Formats • Emphasis is on data dissemination: • Standard data dissemination using an NSDP requiring either a cloud-based Open Data Platform or SDMX web service; • Disciplined publication of data required for surveillance including through observance of an advance release calendar (ARC). • SDMX data registry service IMF IFIs Public; across government agencies; policy makers Data mapping to National Data Portal/NSDP using the ODP in SDMX Country data Data available to the IMF (DSBB) and a wide array of users/partners

7. How we are advancing our systems at IMF • Automated collection leveraging SDMX • eGDDS and ODP • SDMX on NSDP • Global DSD submission though file upload • Machine to machine exchange • IDC • ECB • EuroStat • ….. • Backend Data Processing, Validation, Transformation, Mapping and Integration • External sources • RESTful WS, SOAP WS, RESTful Secure, SOAP Secure

8. Automated collection leveraging SDMX • Automated collection leveraging SDMX • ODP for eGDDS • SDMX maintained by member countries CLIENT (Machine) SERVER (Machine) SOAP Message SOAP Message SDMX Web Service SDMX REQUESTS (IMF) ENVELOPE SDMX RESPONSE (IMF) ENVELOPE SDMX Web Service SOAP Message SOAP Message

9. Design Principles R Reproducible Enable reproducibility of work A Agile Provide for Quick deployment S Scale Provide for Scale D Data Data as a Service C Automate No manual intervention E Elastic Provide for Irregular patterns of use S Secure Provide for Secure environment G Governable Provide for data governance

10. Dissemination Standards Bulletin Board (DSBB) http://dsbb.imf.org/

11. IMF Current challenges • Collect over 70 external datasets daily • From 9 data providers i.e., EuroStat, ECB, OECD, UN, ILO, WB, BIS, US Census Bureau, and Bloomberg • NSDP underlying data links – 689 URL instances; • The web service data is hosted in heterogeneous platforms • Over 25 data consumer: 12 SDDS, 15 eGDDS through ODP • Variety of formats • High data volume

12. A way forward for more collaboration & Partnership • Data Provider: • Make the WS accessible and publicize the service • Allow efficient connection and concurrent data extraction • Engage with IMF Statistics Department • Knowledge transfer • Data Consumer: • Automate the process • Engage with IMF Statistics Department

13. Technology Trends Opportunities