An OPeNDAP server for CORIOLIS data centre

1. An OPeNDAP server for CORIOLIS data centre Context and functions Architecture and technical description Planned improvements T. Loubrieu, A. Forest (IFREMER)

2. DAP4COR : An OPeNDAP server for CORIOLIS data centre • CORIOLIS is an operational data centre for ocean in-situ measurements. • As such CORIOLIS : • Collects measurements for various observation systems (ARGO profilers, measurement from research ships, XBT from Volontary observation ships, moorings, …) • Apply various quality controls on datasets. • Disseminates datasets as asked by users (ocean forecasting systems, ARGO, Carbocean, …) • The datasets are fully managed in an Oracle RDBMS until they are disseminated as netCDF files in ftp directories or requested through a GUI web interface.

3. DAP4COR : external usage requirements • Open a standard interface on CORIOLIS database : • Enable interactive subsetting of the datasets (even with scripts) • Provide an oceanography oriented, high level, interface so that data management and low level information are hidden to the end-user. • Promote CORIOLIS data centre interoperability • Use DAPPER ocean in-situ vertical profile DAP model. • DAP4COR is not intended to be distributed and installed outside of CORIOLIS data centre.

4. DAP4COR : internal requirements • Demonstrate a robust and highly capable federating service for dissemination : • Improve reactivity in the development of new dissemination tools. • Facilitate the maintenance of the multiple dissemination tools by making them independant from the low level RDBMS model. • Avoid datasets redundancy (Oracle RDBMS / netCDF files) to ease data update management. • Better resources management by using a unique RDBMS pool of connexion. • Use J2EE architecture (java + tomcat) because of system infrastructure and available skills.

5. DAP4COR : functions • DAP4COR is Web Application enabling : • OPeNDAP request interpretation for ocean in-situ profile dataset subsetting • OPeNDAP, DAPPER-like response generation from RDBMS profile data. • DAP4COR Dataset configuration (on SQL criteria on the RDBMS native storage). • DAP4COR command line client for saving extraction as NetCDF ARGO profiles file.

6. DAP4COR : request syntax example

7. DAP4COR : user’s subsetting criterias available • Platform code • Geographical area • Time • Vertical reference : depth, pressure • Available measurements • Measurement values ARGO profiling float T/S diagram On a small ocean box Dissolved oxygen Profiles for 2 weeks Mediterranean Salinity profiles One month

8. DAP4COR : dataset configuration • Main data (location, time, measurements) are stored in the same 3 tables. • Ancillary data are distributed into numerous exotics tables (CORIOLIS « expert » domain). • DA4COR Datasets can be configured according to user needs by defining low level SQL criteria on the Oracle DBMS : • Subset of platforms (gliders, elephant seals, moorings array…) • Instrument types (CTD, XBT, XCTD, …) • Quality assesment level (real time, near real time, delayed mode…) • Geographical area (ocean basin, WOCE reference section), by using Oracle Spatial functions (GIS). • … • Thanks to that, a most of the RDBMS model complexity is hidden to the user. • The most relevant datasets are configured on server as SQL pre-defined SQL criteria without data redundancy.

9. DAP4COR : dataset configuration examples • WOCE reference section, for example AX7 • Mediterannean sea • Measurements from elephant seal • Gliders dataset • GTS datasets (BATHY, TESAC messages) • Sea Surface dataset (for satellite calibration) • Pre-processed profiles : pressure/depth conversion, interpolation on standard vertical levels, parameters conversion…  To Be Done • …

10. DAP4COR : response format • Ocean in-situ vertical profiles as proposed in PMEL’s DAPPER server : • One sequence of profiles • For each profile : • First level information (time, location + quality control results) • Metadata (profile attributes) • Sequence of the measured levels (vertical reference value, measured parameters values, quality control information) • Maximum number of vertical profiles in each response : 1000

11. DAP4COR : command line client (dap4cor2netCDF) • Windows/Unix or Linux portable software. • Uses DAP4COR opendap request as input. • Saves the subset as NetCDF ARGO profile file. • Command line tool which can be used as batch (usefull for operational requirements).

12. DAP4COR : system architecture • Components we rely on : • JDAP 1.1.7 • Tomcat application server (enabling load balancing) • Tomcat JDBC connection pool • Oracle RDBMS • Oracle spatial functions (GIS) JDAP 1.1.7 DAP4COR DAP4COR Datasets configuration tomcat JDBC connection pool tomcat application server Oracle RDBMS Spatial functions

13. DAP4COR : a 3 layers software architecture DAP request interpretation/data model generation based on java API (JDAP 1.1.7) In-situ Vertical Profile business interface • Implemented in order to manage : • raw profiles • Or pre-processed profiles (confidential information hidden, vertical reference conversion, interpolated on standard vertical levels…)  To Be Done • Design as a « profile reader » so that memory overflow does not occur. : It reads the lower layer / write the upper layer very frequently so that the smallest possible amount of data is kept is that layer memory. JDBC convenient resultSets

14. DAP4COR : profile business interface • Main methods : • setConstraint() : initializes the OPeNDAP subsetting criterias (dataset, projected parameters, selection parameters) • getStationRoughNumber() – getStationPreciseNumber(): maximum profile quota analysis. • Prepare() : initializes the requested profiles subset (before reading) • Profiles subset reading methods (mapped on DAPPER sequences order) : • nextStation() : move to next vertical profile • getStationElementValue() : get global information on the current vertical profile (time, location, platform, …) • nextLevel() : move to next vertical level for the current profile • nextProfileLevelElementValue() : get and move to next measurement for the current vertical level. • Interface implementation is a parameter of the dataset configuration so that we can define, for example : • A dataset of pre-processed profiles with homogeneous vertical references (pressure/depth converted if necessary) • …

15. DAP4COR : planned improvements • Enable big dataset request (thanks to OPeNDAP indexed requests ?) • Improve performance especially when only the global profile information (time, location, platform code, …) are requested. • Implement alternative profile interface for profile pre-processing (vertical level conversion, interpolation on standard vertical levels…) • Provide a new OpeNDAP interface for other CORIOLIS datasets : • time series at a fixed location • along track observations • Keep on following the DAPPER observation data model convention and improve compliance with it.

16. DAP4COR : links • Server URL : http://www.ifremer.fr/dap4cor • Documentation and related tools : ftp://ftp.ifremer.fr/ifremer/coriolis/tools/dap4cor Containing : • FAQ document • Java client which enable saving datasets as NetCDF ARGO profiles file (dap4cor2netcdf). • Matlab client script examples • Python client script example