Synthesis of the data management activities (NA)

Synthesis of the data management activities(NA) S. Iona (HCMR), R. Lowry (BODC), F. Nast (BSH), D. Schaap (MARIS)

Overview • Training and Capacity building • Metadata Services in V1 • System status and data integration in V1 - EDMED - CSR - EDIOS - EDMERP - EDMO - CDI • Summary

Training and Capacity building • Aims: • to ensure that the data and metadata to be integrated in the system will be formatted, checked for quality and disseminated according to the common protocols developed during the project • to transfer expertise and to train IT experts of the SeaDataNet data centers in the basics, installation and operation of the SeaDataNet technical components

Training and Capacity building • Training Workshops, at IOC Rroject Office for IODE, Oostende, (Belgium): • (Training material and presentations available on SDN Extranet) • 1. February 12-17 2007: • focused on data re-formatting, quality control, and processing using ODV software. • training on project metadata and related web services.

Training and Capacity building • Training Workshops, IOC Rroject Office for IODE, Oostende, (Belgium): • (Training material and presentations available on SDN Extranet) 2. June 4-5 2007: • dedicated on generating XML records with the use of MIKADO tool

Training and Capacity building • Training Workshops, IOC Rroject Office for IODE, Oostende, (Belgium): • (Training material and presentations available on SDN Extranet) 3. June 16-19 2008: • use of the new V1 formats, interfaces and maintenance tools (MIKADO, online CMS, Web services, Validation services, Vocabularies) • data quality control and assessments, using ODV software • analysis and data presentations, using ODV – DIVA software

Metadata Services • One of the SDN Online Services to facilitate the marine data searching, location and retrieval. • The objectives are to: • Maintain and expand the national metadata-bases. • Standardize the information using common vocabularies (as Web Services) • Interconnect the national inventories in a common Pan-European directory.

Metadata Services Structure • It is composed of several thematic inventories of different levels: • High Level Directories – EDMED, CSR, EDIOS (for describing observational datasets collected by ships, by laboratories, by continuous observing systems) • Common References - EDMERP, EDMO and Common Vocabularies (available as Web Services) thatinterconnect the directories • The Common Data Index (CDI) that gives access to the data distributed by the TA platforms (NODCs, Satellite Centers)

System Maintenance – V0 Version 0: 2006-2007 • Continuation and maintenance of past Sea-Search system : • the data access needed several different requests to each data centre • and the data sets were delivered in different formats

System Upgrade to V1 • Version 1: 2008-2010 • Setup of the integrated online data service to users : • networking the distributed data centres • unique request to the interconnected data centres • and the data sets are delivered with a unique format

Maintenance modalities and governance • Maintenance and content upgrade from Version 0 to Version 1 by using on-line (like CMS forms) and off-line tools (MIKADO) that produce XML ISO 19115 compliant exchanges and developed by the Technical Task Team in the joint research activities JRA1, JRA2: Reformatting Tools now available: - Med2MedSDN - NEMO Generation ofcoupling tablefor DM, mapping download: - NEMO - Mikado V1.6 Training hour tomorrow

System status and data integration in V1 • European Directory of Marine Environmental Datasets (EDMED) • Cruise Summary Reports (CSR) • European Directory of the initial Ocean-observing Systems (EDIOS) • European Directory of Marine Environmental Research Projects (EDMERP) • European Directory of Marine Organizations (EDMO) • Common Data Index (CDI)

European Directory of Marine Environmental Datasets (EDMED) • V0 to V1 content changes - Content developed to be ISO19115-compliant through a profiling exercise • Major change was that fields such as parameters were converted from plaintext to vocabulary term lists that could be implemented as ISO19115 keywords • Some fields needed for mandatory ISO19115 core were added • A small number of fields rendered redundant by plaintext replacement were dropped

European Directory of Marine Environmental Datasets (EDMED) • Content conversion plan • A CMS has been provided by Maris (http://www.seadatanet.maris2.nl/vu_edmed/welcome.asp) - Partners upload V0 records into the CMS - CMS template guides conversion to V1 - V1 records exported as XML and ingested into BODC database

V1 Infrastructure Status • Infrastructure Work Completed - V1 Oracle storage schema - V1 XML ingestion software - V1 Web Service export • Infrastructure Work To Do - Upgrade presentation schema to V1 - Storage to presentation schema translator - V1 User Interface

V1 Content Conversion Status • BODC - 703 (out of 3357!) V0 records are the responsibility of BODC • Initial conversion completed by junior member of staff • List of issues to be resolved by Lesley - Status of other partners’ progress checked on 2009-03-23

V1 Content Conversion Status Countries with entries completed or nearing completion

V1 Content Conversion Status Countries with entries apparently not converted

V1 Content Conversion Status • SUMMARY • Records upgraded (as of 23 March): 1448 (45%) • Countries supplying V1 records in XML (i.e. will not be converted in CMS): France, Germany, Ireland, Italy, Netherlands • CLS (France) will contribute with V1 records in XML • Romania, Portugal: large collections with no upgrade or submission of XML files • Remaining records to be upgraded: ~430 • (excluding BODC’s 700) • Upgrade needs to be done very soon, assistance is available

System status and data integration in V1 • European Directory of Marine Environmental Datasets (EDMED) • European Directory of Marine Environmental Research Projects (EDMERP) • Cruise Summary Reports (CSR) • European Directory of the initial Ocean-observing Systems (EDIOS) • European Directory of Marine Organizations (EDMO) • Common Data Index (CDI)

EDMERP content upgrade • Using CMS for upgrading V0 to V1 and also for adding new records • Using MIKADO and XML import • [available as webservice for XML V1 files (export and import)]

http://seadatanet.maris2.nl/v_edmerp/search.asp Browse list Additional details • EDMERP Web Interface - Retrieval

http://seadatanet.maris2.nl/vu_edmerp/welcome.asp • EDMERP CMS

STATUS OF EDMERP UPGRADING FROM V0 TO V1 LEVEL, 17-Mar-2009- 95% of the upgrade is done- the rest should finish the next months

CSR Status • ~ 40 000 CSRs (1873 -2009) • in an Oracle DB • 17 partners submitting new entries in V1 • 16 entries from 2009 (Skandinavia)

V1 Maintenance • New entries (and updates) • Online for individual entries (CMS) • XML files for bulk submission with the use of MIKADO tool

CSR Content upgrade to V1 • - Online • contact BSH-DOD requesting CSRs to be updated • BSH-DOD loads requested CSRs to entry database • modify CSRs in entry database • - Offline • contact BSH-DOD requesting CSRs to be updated • BSH-DOD sends requested CSRs as XML V0 files (free text entries) • Use of MIKADO tool and submit modified CSRs in XML V1 format to BSH-DOD

Status of Upgrading to V1 • 7 Partners upgrading systematically using online application • 17 partners submit new online entries in V1 • 5 partners submit new entries and upgrades in XML V1 • Slovenia (NIB), Italy (OGS), UK (BODC), Turkey (IMS-METU) and France (IFREMER) • altogether around 4800 completed CSRs in V1 format • only 10% is in V1 !!! • Remark: • many project information which were originally in the V0 format get lost due to missing EDMERP entries

big new entries and upgrades of XML V1 submissions by 5 partners: France, UK, Italy, Turkey, Slovenia • operational new online entries in V1 by 17 partners

ACTION Suggestion • Begin with the latest data and go backwards • Partners get export to update those CSRs either online or by Mikado • Partners get a reminder, may need help? ISSUE • only part (10%) of the total CSRs are now upgraded, mainly from the last decade • not all partnersstarted upgrading • not all partners started using new online entries in V1

New features in CSR User Interface • CSR Online Entry - uploading of track charts, station lists and additional reports directly into database • Organisation link to EDMO record • Platform link to EurOcean RV Infobase • XML download in V1 format for upgraded CSRs (being tested)

CSR V1 Retrievalhttp://seadata.bsh.de/csr/retrieve/V1_index.html EurOcean RV Infobase link EDMO link Track chart

CSR V1 CMS http://seadata.bsh.de/csr/online/V1_index.html Upload station list Upload reports Upload track charts

CSR V1 Retrievalhttp://seadata.bsh.de/csr/retrieve/V1_index.html

Interconnecting Meta-catalogues: Link from CSR to EDMERP Link from EDMERP to CSR to CDI ? • Printable form for CSR • Planning CSR Web Service for use together with MIKADO • Use of POGO ship schedule information for preparing initial CSRs Planned Actions

Status • No new input has been requested over the last year • Improved Oracle (database) schema • Supported by common vocabularies, EDMO and EDMERP • ISO 19115 content model: • Description of the format and XML tags available • XML example file available • Programme and Series XML Schema available • Upgrade is underway • MIKADO V1.6: for new entries and updates • New user Interface available

New User Interface: http://seadatanet.maris2.nl/v_edios/search.asp

Maintenance • Update of existing entries via CMS forms • Access for NODC‘s • New records only to be created via MARIS (because EDMO = vocabulary) • Available as webservice (export only)

European Directory of Marine Organizations • Currently, EDMO lists and describes more than 1.500 organizations. • There is direct crosslink with EDMED - data sets, EDMERP - research projects, CSR - research cruises, EDIOS - observing stations/ systems, and CDI V0 and V1 - index to data. • Content upgrade from V0 to V1 is finished

EDMO V1 search and retrieval http://seadatanet.maris2.nl/edmo

EDMO CMS http://seadatanet.maris2.nl/vu_organisations/welcome.asp EDMO CMS geo-locator via Google maps

Common Data Index: gate to data access • CDI V0 System: • More than 340.000 CDI entries - Data requests through different requests at data centres - Data delivered at different formats • CDI V1 System: • Unique request • order and access for downloading via a shopping basket mechanism and a communication between the SeaDataNet portal and the local systems • Data delivery in unique format

CDI V1 – Quick Search http://seadatanet.maris2.nl/v_cdi_v1/browse_step.asp

CDI V1 – Extended Search http://seadatanet.maris2.nl/v_cdi_v1/search.asp

Integration of data centres in CDI V1 • 9 partners fully connected, 386.743 CDI files loaded in the production environment • 15 partners waiting the DM/interim tests, 130.379 CDI files loaded in the import system • 18 partners still to come, assistance is available

Summary • The system upgrade from V0 to V1 is making good progress, but needs to be completed very soon and especially CDI (considering that the EMODNET chemical pilot is starting) :

Synthesis of the data management activities (NA)