JERICO WP5 Data management and distribution

1. JERICO WP5 Data management and distribution Alessandro CriseIstituto Nazionale di Oceanografia e di Geofisica Sperimentale JERICO Project Kick-Off meeting Maison de la Recherche Paris, 24-25 May 2011

2. JERICO WP5Data management and distribution: main goal The main goal is to design the JERICO Data Flow and Management infrastructure based on the implementation of an end-to-end ‘open and free’ conduit from data to users and upon the guidelines proposed in the GOOS Report 148 (2005). The implementation will be consistent and complementary with the major European initiatives for the establishment and coordination of infrastructures for the management and distribution of data and products in Europe which can use JERICO data (SeaDataNet, EuroGOOS Regional Centres, MyOcean). YET ANOTHER DATA MANAGEMENT SYSTEM?: NO!

3. WP5 Structure • JERICO will need to ensure that the flow of real-time and delayed mode will be reliable, accessible and easy to distribute. • The WP5 structure: • • Creating value for measured data: from raw data issued from the sensors to validated data stored and marked. • • Facilitating data and meta-data flow to existing data repository infrastructures (SeaDataNet); • • Helping in delivering real-time JERICO data to ROOSs and My Ocean

4. Task 5.1: Quality assurance: create value for measured data • Task responsible: OGS • Partners: HCMR • Duration M1-M42 • This task is aimed at adding value to observations by associating them with characteristic uncertainties. • By definition, the “expected” accuracy associated with an observation is an attribute of the measurement chain that produces it.

5. Task 5.1: Quality assurance objectives • The objectives are: • Collecting pre-existing knowledge regarding the key identified sensors and critically revise state-of-art literature on in lab and in situ performances of the most popular sensors (in conjunction with WP4) • defining procedures for ascribing a total uncertainty to measurements that will account for the actual circumstances and conditions under which the measurements were taken. • Testing and verifying uncertainty computational procedures

6. Task 5.1: Quality assurance methodologies • . Methodology • A novel model of a-priori uncertainty will be proposed as a sum of fixed and time-dependent source of error and uncertainty (precision and accuracy, drifts, experimental uncertainties, the representativity error, etc.) • The variables temperature, salinity and fluorescence/chlorophyll will be considered at first. • Controlled experiments will be performed in order to document and validate the different uncertainty components composing the final uncertainty estimates. • All the relevant assessments carried out in WP4 will be included as basic information for the combined uncertainty estimation. • WP4.1 will provide an updated summary of these achievements to WP5.1, no later than the dates set down in the WP4 D4.2 delivery schedule (M14)

7. Task 5.2: Harmonization of Delayed Mode data management procedures with SEADATANET • Task responsible: IFREMER, • Partners: HCMR, MUMM, OGS • Duration: M1-M42 • This task will define, establish and oversee the data management infrastructure for dealing with delayed-mode data in JERICO. • The goal is to provide JERICO users with relevant informations for seamless data integration with the EU SEADATANET initiative.

8. - Survey the existing delayed-mode data handling practices of JERICO partners. • Formulate a viable proposal for a common JERICO delayed-mode data management platform that can serve to reinforce the SEADATANET effort; • the guidelines established by DG-Mare/EMODNET and WISE-Marine will be • also taken in account when doing this. • - Create common vocabularies for JERICO delayed-mode data formats and meta-databases, building upon the work that has already been done in SEADATANET. • Implement delayed-mode data and metadata formats that are compatible with those of SEADATANET (in compliance with the EU INSPIRE Directive); • the Sensor Web Enablement family of the OGC consortium family of standards (SensorML, Sensor Registry, O&M) will be considered in the implementation. • - Ensure easy sharing and secure archival of delayed-mode data within JERICO by employing common data transport formats and storage criteria making use of the experience gained by SEADATANET. Task 5.2 Methodology

9. Task 5.2 Metadata implementation Discovery metadata To let all know what have been deployed during the project (where, what, how, who…) To help to retrieve data related to observatories To be compliant with European directives (Inspire) Enabling technology Use of SeaDataNet EDIOS directory to describe observatories Typing / Generation of metadata using SeaDataNet procedures and tools (Mikado) complemented with automatic tools developed in Emodnet-Physics to better use the RT Netcdf files provided in RT by the ROOSs

10. Task 5.2 Data implementation Delayed mode data Delayed mode quality check Uniform file format (based on OceanSites NetCDF, compatibility with CF convention)+ distribution in ASCII flat file format (Ocean Data View) Full metadata (Common Data Index) generated by Mikado for long term reuse Data access via web portal (all users)

11. Task 5.3 Harmonization of Real Time data anagement procedures with MyOcean and EuroGOOS & the institution of data access services for JERICO TOP activities • Task responsible: IFREMER, • Partners: CNR, NIVA, IMR, HCMR,PUERTO, SMHI • Duration: M1-M42 • This task will define, establish and oversee the data management infrastructure for dealing with real-time data in JERICO. • The goal is to provide for seamless integration with the EuroGOOS and EU Marine Core Service’s MyOcean component and in order to render JERICO real-time data accessible for use in generating downstream services and products.

12. Task 5.3 Methodology • Survey the existing real-time data handling practices of JERICO partners. • Formulate a viable proposal for a common JERICO real-time data management platform that can serve to reinforce EuroGOOS and MyOcean • Create common vocabularies for JERICO real-time data formats and metadatabases, based on any existing • recommendations by MyOcean/EuroGOOS for doing this. • Implement real-time data and metadata formats based on any existing recommendations by EuroGOOS and MyOcean for doing this. • Reconcile, wherever possible, the data quality assurance procedures/protocols for real-time data amongst the JERICO partners (link to WP4, WP5.1). • Ensure easy exchange of real-time data within JERICO by employing common data transport formats and storage criteria, based on any existing recommendations by EuroGOOS/MyOcean for doing this.

13. WP5.3 Implementation Real-Time data Real-Time Quality check based on EuroGOOS/MyOcean procedure that would need to be extended to new parameters whenever possible Uniform file format (based on OceanSites NetCDF, compatibility with CF convention) ftp file server (with an index file)per region Oceanotron (OpenDap protocol) for direct access by remote software

14. WP5 in JERICO Data flow WP5 act as mediator in JERICO Data Flow Model A priori quality assurance (WP4) Data management procedures (WP5) Time-dependent data uncertainty estimation Real-time data management Delayed mode data management Value added products and data dissemination (WP6) EuroGOOS ROOSs engagement (WP2) Demonstation of data provision (WP7) SeaDataNet MyOcean Emodnet Physical Parameters EuroGOOS

15. MyOcean Insitu TAC implementation of EuroGOOS regional approach Arctic: IMR / Norway All parameters • Baltic Sea: SMHI/Sweden • Temp & Salinity : BSH/Germany • Current: SMHI/Sweden • Sea Level: DMI/Denmark • Bio : FMI-Syke/Finland • North West Shelves: BSH/Germany • Temp & Salinity : BSH/Germany • Current: SMHI/Sweden • Sea Level: DMI/Denmark Black Sea IOBAS/Bulgaria All parameters • Global Ocean : Coriolis/France • Link with international network s: Coriolis/France • Argo: Coriolis,BODC,BSH • European Vessels: NIVA/Norway v • Mediterranean Sea : HCMR/Greece • Mooring : HCMR/Greece • XBT/CTD: ENEA/Italy • Drifter & Argo: OGS/Italy • Glider and Argo: Coriolis/France • South West Shelves: Puertos Del Estado/Spain • Mooring : PdE • Underway data: Coriolis/France

16. MyOcean In-situ TAC role Limited number of parameters: Physical: T&S, current, sea level Biogeochemical: Chlorophyll/Fluorescence, Oxygen, Nutrients Integrate in-situ data in product accessible through global and regional portals Common format Common NRT QC Common Quality flags Common distribution tools Single access point Ensure a minimum level of quality on the data delivered In Near real time ( 24h to a week) In delayed mode Assessment of the products at basin scale

17. The European Commission, represented by DG MARE, concluded a service contracts aiming to: Create a prototype components of the European Marine Observation and Data Network (EMODnet). Provide access to archived and real-time data on physical variables in Europe's seas and oceans determine how well the data meet the users needs. EMODNET PHYSICAL PARAMETERS

18. THE EMODNET PP PILLARS EMODnet PP JERICO WP5 EMODnet PP aims to bridge the communities behind MyOcean, EuroGOOS and SeaDataNet in order to strengthen their position inside the Commission agenda on IS and expand in future the participation to all partners SeaDataNet MyOcean EuroGOOS

19. EuroGOOS SEPRISE network NEW FEATURES • Add new stations and models • Add profiles • Make use of the results from ECOOP • Add data from FerryBox and RV • Common quality control • Link to EDIOS • SEPRISE data in SDN ODV format • Put data on GTS • E-HYPE

20. Cooperation between MyOcean in-situ TAC and SeaDataNetMoU signed Beginning 2010 Operational users Marine governmental agencies and research institutes (including NODCs) MyOcean in-situ TAC Direct users real-time Re-analysis users Observing systems real-time Data integration (QC, « best copy », archival, re-analysis datasets Vocabularies Metadata (EDIOS, CDI, CAMIOON, SensorML) Technical WG Metadata (e.g. EDIOS) Yearly regional products (climatologies, datasets) Other observations(e.g. scientific cruises…) delayed-mode SeaDataNet Users International organisations (IOC/IODE, ICES, ICSU World Data System…) Direct users

21. JERICO role in the European Data Management landscape ‘EMECO-like’ Web – enabledDatatools (WP6) Web Portal Observing Network Assessment database ‘EMECO’ like Data XML, CSV Operational users JERICO Coastal Observatory Network (WP7) MyOcean in-situ TAC real-time Re-analysis users real-time Data integration (QC, « best copy », archival, re-analysis datasets Vocabularies Metadata (EDIOS, CDI, CAMIOON, SensorML) Technical WG Information & evidence pipeline Metadata (e.g. EDIOS) Yearly regional products (climatologies, datasets) delayed-mode SeaDataNet Users International organisations (IOC/IODE, ICES, ICSU World Data System…) Direct users

22. WP5 Milestones/Deliverables

23. ACTIONS • To propose a formal agreement with SDN, EuroGOOS/MyO for JERICO data provision, to ensure more visibility to JERICO project • To find an agreement with DG MARE/EMODNET for the availability to JERICO of the developments achieved/to be achieved in EMODNET Physical Parameter lot.

24. Thanks!