WMO RA VI Workshop on the implementation of WIS Sofia, 1-3 November 2011

1. The MeteoSwiss Data WarehouseReal-time access to data and metadata – basis for the WIS-implementation in Switzerland WMO RA VI Workshop on the implementation of WIS Sofia, 1-3 November 2011 Stephan Vogt, Meteorological Data Coordination, MeteoSwiss stephan.vogt@meteoswiss.ch

2. Agenda • Situation in 1999 • Towards the MeteoSwiss Data Warehouse • Metadata in the DWH • Ongoing improvements • Considerations on the WIS-implementation

3. Situation in 1999 – and the vision Year 2000 Problem: maximum credible data accident to be expected?

4. Objectives of the MeteoSwiss Data Warehouse • Implementation of a central, integrated data platform for the collection, storage and processing of meteorological data and its metadata. • Storage of metadata in one central metadata repository • Subdivision in a processing and an application area • Creating a basis for a customized access to data respecting official standards for internal and external clients

5. 2000 Start of the MeteoSwiss Data Warehouse Project 2001 Surface data (rain gauge measurements, AWS data, observations) 2002 2003 Phenology & Pollen 2004 Aviation messages 2005 Lightning data Metadata 2006 International surface data (SYNOP) 2007 National sounding measurements 2008 Homogenized Data 2009 National & International Upper Air Data 2010 Gridded Datasets 2011 Model Data

6. T L Conceptual Architecture Climate Information Service Model Validation Research Climatology Data for the Public Internet Data analysis Durable data archive Weather Forecast Data delivery and products (on-/offline) Data analysis & Interfaces (API, DB Tools, Web access, GIS, R …) Kontextdaten Database for point data analysis Data storage Geodatabase(raster data, ...) Meta Data Metadata Management Tools Database for point data processing (incl. raw data repository) Staging area QC Calc, Check, Data Transformation & Integration Data sources Partner Observ.systems Autom. Observ. systems Light- ningdata ClimaAnalysis Tools UpperAir data Obser- vations Aviation data CM SAF NWP VERA Radar

7. Metadata in the MeteoSwiss DWH …stored in more than 200 tables in the DWH

8. customers Raw Data Clean Data Quality control Instrument Lifecycle Homo-genization measurement simulation homogenization quality control II calculation of derived quantities quality control III errorr correction data collection quality control I Station history Enhanced Data Aggregation calculation Spatialreference Contractdata Data import immediately extreme value messages Parameters level 2 data level 1 data level 3 data data source level 4 data level 5 data immediately Raw data repository days years applications for data distribution Data repository for external data external data source applications for data processing Legend Level 1 measurement data Level 2 raw data Level 3 cleaned data Level 4 corrected data Level 5 homogenized data metadata durable data archive MeteoSwiss data archives durable data archives outside MeteoSwiss MeteoSwiss data sources data sources outside MeteoSwiss data transmission and distribution

9. Master Metadaten Repository Development cycle & metadata management Development Production Deployment Failover Production

10. How are the metadata accessed? • DWH-Webtool with user guidance for all important use cases • MS ACCESS for all other cases

11. Look and Feel of DWH-Webtools

12. Advantages of the Data Warehouse • measurement data • metadata • control data • quality tests  Allows an efficient and economic maintenance  company-wide data integration platform for the long term  independant from technical realisation specified in a uniform way, centrally administrated and configurable

13. Lessons learned

14. Do things the same way all over the organisation  data governance • Data architecture documents • Gremium with representatives from all MeteoSwiss departements

15. Not everything is solved... • RENAISSANCE = Renewal and ENhAncement of the servIceS baSed oN the mEteoswiss data warehouse • Subprojects: • Data supply of COSMO from DWH • Generic delivery mechanism for data marts • Station history • Consolidation of production tools

16. Generic delivery mechanism for data marts • Existing delivery mechanisms have been generalized • Towards event-based triggering of processes • Generic delivery mechanism includes • Measurement data • Metadata • based on XML standard  Primary purpose: Data delivery to GIN (Swiss joint Information Platform for Natural Hazards)

17. Lots of tools for lots of products…

18. …and what about WIS? • All remarks so far refer to our status as NC • MeteoSwiss has been asked to develop GAWSIS as a DCPC for WIS • Today GAWSIS is not yet integrated into the DWH • Two options for the next steps: • solution on the basis of the current GAWSIS  Provide the data with an OAI-Service  ready for collection by GISC • Integration of GAWSIS into the DWH, make use of the planned Web Service-infrastructure

19. Vision regarding WIS

20. Vision regarding WIS

21. Questions? Thanks for your attention! stephan.vogt@meteoswiss.ch