Francesco Rizzo (ISTAT - Italy)

1. SDMX ISTAT FRAMEWORK Francesco Rizzo (ISTAT - Italy) GENEVE 10 -11 May 2007 OECD SDMX Expert Group

2. Summary SDMX Istat Framework Why Istat is starting to use SDMX SDMX Istat Framework – an overview SDMX data Web Service (demo) SDMX Web Navigator (demo) Geneva 10-11 May 2007 OECD SDMX Expert Group

3. Production Directorates Data Collection Data Editing Data Aggregation thematic DB metadata web navigator web navigator Why Istat is starting to use SDMX SDMX Istat Framework The Istat Information System is made up of several production Directorates sub-systems that, independently one from another, cover the full life cycle of statistical data, from collection to dissemination. Geneva 10-11 May 2007 OECD SDMX Expert Group

4. metadata Integrated Output Management System project Production Directorate Production Directorate Data Collection Data Collection Data Editing Data Editing Data Aggregation Data Aggregation validated microdata thematic DB thematic DB Integrated Output Management System web navigator web navigator web navigator web navigator Geneva 08-10 May 2007 Meeting on the Management of Statistical Information Systems

5. Why Istat is starting to use SDMX SDMX Istat Framework For this reason Istat is following the evolution of the SDMX initiative with interest: Tactically: through the participation in the EUROSTAT SODI task force Strategically: through the building of a special working group whose main objective is to analyze and verify the use of SDMX in the internal architecture of ISTAT Information System Geneva 10-11 May 2007 OECD SDMX Expert Group

6. Why Istat is starting to use SDMX SDMX Istat Framework The participation in the EUROSTAT SODI initiative has involved Istat to gain experience in the use of XML and Web Services technologies for integration purpose inside Statistical Information Systems. In order to facilitate this objective, we are developing several software modules organized in a framework (current version is 1.0) Geneva 10-11 May 2007 OECD SDMX Expert Group

7. SDMX Istat Framework version 1.0 SDMX Istat Framework • Characteristics: • compatible to SODI needs • useful to introduce SDMX for both internal purposes and external information exchange • can be used entirely from the reporting phase to the dissemination phase, or alternatively using modules separately, integrating them into one information system • in the future it might be possible to distribute the framework under Public Licence Geneva 10-11 May 2007 OECD SDMX Expert Group

8. Reference metadata RSS Reference metadata manager Reference metadata SDMX Query Reference metadata web navigator Reference metadata web service Reference metadata database IMF STS Eurostat Word extractor ESA SDMX web navigator mapping STS-ESA time series database SDMX data web service .ist gesmes ConIstat web navigator RSS builder & query builder Loader .dat Check SDMX Queries SODI RSS ConIstat modules new modules fix formatted record file under construction modules GENEVE 10-11 May 2007 OECD SDMX Expert Group

9. SDMX Istat Framework version 1.0 SDMX Istat Framework • Overview of changes in the Loader module: • New dissemination functions were added in order to satisfy some SODI requirements: • to publish an RSS; • to publish on or more SDMX Query file(s); • to publish one or more SDMX Compact file(s) (optional) Geneva 10-11 May 2007 OECD SDMX Expert Group

10. queryParser sqlquery builder XML validate SDMX Query SDMX Compact SDMX Compact builder Database SDMX Istat Framework version 1.0 SDMX Istat Framework Overview of the SDMX Data Web Service module: Response-time test: an entire Dataflow containing 685 time series, 120 months long (developing 82.200 observation) < 20 seconds demo Geneva 10-11 May 2007 OECD SDMX Expert Group

11. SDMX Istat Framework version 1.0 SDMX Istat Framework • Overview of SDMX Web Navigator module: • This module acts as a client towards the web service and represents a web graphic interface that allows the following functions: • building a SDMX Query using DSDs as analysis dimensions • sending the SDMX Query to the web service for querying the database • Testing the SDMX Queries that a user can build starting from SDMX Queries created with the graphic interface • demo Geneva 10-11 May 2007 OECD SDMX Expert Group

12. Database Production Directorates Reference metadata manager XSLT XML Reference metadata web navigator HTML Reference metadata web service SDMX Word extractor Eurostat IMF SDMX Istat Framework version 1.0 SDMX Istat Framework Overview of Manager and Navigator SDMX reference metadata module: Geneva 10-11 May 2007 OECD SDMX Expert Group

13. SDMX Istat Framework version 2.0 SDMX Istat Framework • A generalized SDMX Query parse module (beta version): • This module accepts only queries with DataWhere section but also accept queries regarding: • different Dataflows • subsets of different DataFlows • A SDMX Query is processed by the following steps: • rewrites the XML stream using Boolean logic rules, and adding to each node an attribute that acts as a “unique key”. The scope of the unique key is to set a hierarchy between “parent nodes” and “child nodes”; • converts the XML stream in a memory tabular data structure whose columns represent all types of nodes (Time, Dimension, Attribute, Dataflow) except for And and Or; • re-organizes the in-memory tabular data structure so that all columns have an And relation, and all rows have an Or relation; • converts the in-memory tabular structure in a SQL query Geneva 10-11 May 2007 OECD SDMX Expert Group