Quoi de Neuf à Coriolis en 2008 ? GMMC 13-15 Octobre 2008 S Pouliquen & Coriolis team

1. Quoi de Neuf à Coriolis en 2008 ?GMMC 13-15 Octobre 2008S Pouliquen & Coriolis team Developing a Data Management System for Glider data in Europe Sylvie Pouliquen GROOM Annual meeting 4-5 June 2013

2. The Objectives of an integrated data system • Data accessible easily from a unique point • Data coherent in term of : • Data format • Data Quality • Processing chain ( clearly documented) • Serve both Operational and Research Users • Data are available in near real time ( within less than 24 hours) • Data are available in delayed mode after calibration and /or validation

3. What About Gliders • Glider activities are presentlydriven by individual research drivers • Gliders can deliver real-time data for core parameters( T, S Current , Chl, O2) that are useful for both research an operational users • Benefit from what has been developed by Argo and OceanSites • Developing integrated Data Management system • Common Data format to users • Real –time QC of core parameters • Gliders are complementary to other platforms and synergyshould be developed • Developing a deployment strategy for other needs than pure research ( ie. GMES Marine Core Service in Europe)

4. What is the starting point for GROOM • Data management is done by the different research communities using their own methods • Some harmonization activities started in FP5-MFSTEP and FP6-MERSEA projects and FP7 MyOcean • Data exchange in Real time is working on a best effort schema through EGO but without any commitment nor from providers or data managers • Link with GMES MCS is done though Coriolis providing Glider data as profiles • No agreement neither on RTQC or DMQC but best practices on RTQC through MyOcean INSTAC not widely known by the glider teams

5. Groom Data Management activities Task 3.2

6. GROOM Data Management Objectives • Data accessible easily from a unique point • Data coherent in term of : • Data format • Data Quality • Processing chain ( clearly documented) • Serve both Operational and Research Users • Data are available in near real time ( within less than 24 hours) • Data are available in delayed mode after calibration and /or validation

7. Objective of task 3.2 • Improve coherency of Glider dataset in Europe • Facilitate access to Glider data by defining a Data system for Glider data • The system is built to exchange glider data • Must first be useful for Glider users and Glider operators and therefore provide access to as much as possible information (metadata, scientific, technical data) provided by Gliders • Delivering of Glider Data either on GTS or FTP for Operational users ( i.e. MyOcean) will be carried on as specific delivery system that will convert the GDAC data into products usable by these operational users

8. Who does what ?

9. GROOM Data flow PIS Final correction Data to DAC - GDAC ScientificUsers Operational Centers GTS FEED GDAC France RT withinhours RT DATA QUALITY CONTROL Data assembly Centres GliderOperators DAC DAC DAC DAC Withinhours

10. Roles of Glider Actors • PI : Principal Investigators • Team orscientists who define the glider mission, deploy the glider and carry out post-recovery delayed mode QC • Glider Operators • Team in charge of steering the glider, collecting all the metadata and the deployment information required for processing, collect all the data transferred in realtime by the glider. Collect the post-recovery high resolution data

11. Roles of Glider Actors • the Dacs : the DAC is the facility set up by one or more nations/institutes to provide RealTime and Delayed mode glider data to the users. It • collects the data from the Glider Operator, • converts to standard exchange format, • applies standardized real-time quality control, • delivers data to the GTS and GDACs within few hours of the surfacing and to PIs , • coordinates glider data handling for the gliders under their control.

12. List of DACs

13. Roles of Glider Actors • GDAC : The GDAC operates the data services where the master copies of the data resides. It doesn’t perform any additional individual glider QC activities. • Central point for data distribution on Internet for all GROOM gliders • Can perform data format transformation, of set up additional services ( OGC viewing service, OpenDap/Oceanotron download services ,…) to fulfil additional needs.

14. Common data format

15. Improve Coherency of Glider dataset in Europe • Worked on the definition of a Glider , of a deployment for a glider. • A glider is moving platform that is steerable. It can have a propeller and this information must be recorded in the metadata. • Define Level of processing: • Level0 : Data provided by the glider without any unit transformation or geophysical interpretation. • Level1: Geophysical parameters with a quality indicator set up by automatic QC procedures together with the data acquired by the glider. This is the level shared in Near Real Time. • Level2 : Geophysical parameters calibrated after glider recovery together with quality flag information, if possible error estimation together with the non-corrected data provided at Level1. This is the level shared in Delayed Mode • Level3 and after : Product derived from glider data ( gridded fields, additional parameters calculated …) This is not addressed in the present GROOM data management activities.

16. How to Describe a Glider Sensor S Sensor S Sensor S Sensor S Takes on board Performs Deployment N Description of the glider for this deployment • What are the sensors on board • What is the configuration Deployment N Deployment N Is described by Deployment N Transmits for each dive Scientific measurements Scientific measurements Scientific measurements Technical information measurements Scientific measurements Technical information measurements Scientific measurements Technical information measurements Technical information measurements Technical information measurements

17. Improve Coherency of Glider dataset in Europe • Worked on common data format to share the Glider data based on OceanSites NetCDF format already used in EuroGOOS/ MyOcean/ SeaDataNet • Defined how to store Metadata to register the mission description and its evolution in time when changed through down link • Enhanced OceanSites format for Scientific information by adding if necessary the new parameters sampled • Defined how to store the technical information by definition common vocabulary • First version of the user manual was delivered • Common tools to produce these Netcdf files are under way

18. RT Processing chain modules Slocum Seaglider Other gliders Source data GenericDescription • Platform • Sensors Common Readers in matlab that convert Manufacturers Data into .mat files My Glider description Real Time quality control routines Processing modules Conversion of .mat output to EGO format NetCDF for transmission Data delivery PI GDAC GTS

19. Data Stream in Real time Manufacturer Format DATA Common Format Glider Operator Customize to meet other needs GTS Users DATA Sci Data MyOcean Users Sci and Tec data Sci and Tec GDAC users DAC

20. Define Real Time QC procedure • Action taken to work on common Near Real Time QC procedures in agreement with what exist already within EuroGOOS/MyOcean/SeaDataNet • Adopt EuroGOOS/MyOcean NRT QC procedures for T&S, Chl and O2 • Enhance if necessary these procedures to take into account Glider specific behavior • Develop new recommendations in Partnership with Myo/SDN for additional parameters available in real time

21. Data Stream after recovery Manufacturer Format Common Format High Res Manufacturer Format HR High Resolution scientific data Glider Operator Real Time Data in Common format ----------Delayed mode High resolution data GTS Users Customize to meet other needs HR TEC Sci Data MyOcean Users Common Format Common Format GDAC users DAC

22. Post recovery procedures • The Goal : correct the data transmitted in real time • Check RTQC again. Additional run taking into account past and future data is possible (future data not available in RT !) • If necessary Merge RT data with flash card data in order to fill the gaps left by the RT data transmission in terms of resolution and/or parameters measured • Cross-calibration along the whole deployment with reference to in-lab water samples measurements when possible. • Working group to develop recommendations for T&S, Chla, Oxygen, Current

23. Possible extention outside Europe through the EGO Cost action

24. GROOM Data flow PIS Final correction Data to DAC - GDAC ScientificUsers Operational Centers GTS FEED GDAC France RT withinhours RT DATA QUALITY CONTROL Data assembly Centres GliderOperators DAC DAC DAC DAC Withinhours

25. Possible extention outside Europe PIS Final correction Data to DAC - GDAC ScientificUsers Operational Centers WWW FTP GDAC France GDAC ? GTS FEED Archive? RT within hours Glider Info Center? RT DATA QUALITY CONTROL Data assembly Centres GliderOperators Monitoring DAC DAC DAC DAC Withinhours

26. Data Format and QC procedures • Data Format : based on OceanSites netcdf format with CF convention also used by IMOS /Australia • Compare the format metadata to be sure that we are coherent (link with ODIP EU project): • name the same thing the same way • list of mandatory fields that should be in all files • Share tools to generate netcdf files • NRT QC procedures : Both IMOS and QUARTOD procedures are taken as input for defining the EU RTQC procedures for core parameters. • Post-recovery procedure : still and R&D activity but collaboration with international partners should be encouraged

27. Quoi de Neuf à Coriolis en 2008 ?GMMC 13-15 Octobre 2008S Pouliquen & Coriolis team Thank You Questions ?