420 likes | 578 Views
SeaDataNet - Pan-European infrastructure for ocean and marine data management Present state, relations to EMODnet, and future developments By Dick M.A. Schaap – MARIS (SeaDataNet Technical Coordinator) Copenhagen – Denmark, May 2011, EEA – WISE-MARINE Meeting
E N D
SeaDataNet - Pan-European infrastructure for ocean and marine data management Present state, relations to EMODnet, and future developments By Dick M.A. Schaap – MARIS (SeaDataNet Technical Coordinator) Copenhagen – Denmark, May 2011, EEA – WISE-MARINE Meeting Supported by the European Commission
EU FP6 2006 - 2011 SeaDataNet has set up and is operating a pan-European infrastructure for managing marine and ocean data by connecting 40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europe
Interoperability Interoperability is the key to distributed data management system success. This is achieved in SeaDataNet via: • Adopting the ISO 19115 metadata standard for all metadata directories • Providing standard metadata entry tools: MIKADO, and online CMS • Providing XML Validation Services to quality control the metadata maintenance • Using common and controlled vocabularies, including international content governance • Using harmonised Data Transport Formats for data sets delivery and standard tools for format conversion, such as NEMO, .. • Using common quality control protocols and flag scale • Adopting of OGC standards for mapping and viewing services • Using SOAP Web Services in the SeaDataNet architecture • Organising training and instruction workshops for transfer of expertise and hands-on training
Portal with harmonised services, data products and tools for users and data centres http://www.seadatanet.org
Pan-European Directory Services The SeaDataNet portal provides overviews of marine organisations in Europe and their engagement in research, scientific cruises, monitoring and data management for European waters and global oceans: • EDMO: European Directory of Marine Organisations (>2200 entries) • EDMED: European Directory of Marine Environmental Data sets (>3000 entries) • EDMERP: European Directory of Marine Environmental Research Projects (>2500 entries) • CSR: Cruise Summary Reports (>31500 entries) • EDIOS: European Directory of Ocean-observing Systems (>270 programmes for the UK alone and many underway for other European countries) These Directories are maintained by NODCs for their country and published at pan-European level
Pan-European Directory Services All these Directories have been harmonised and mutually tuned in format, syntax and semantics (common vocabularies), a common XML editor (MIKADO), online CMS and online user interfaces. EDMERP CSR EDMO EDIOS EDMED
Relations between Directory Services From the EDMO interface one can oversee and retrieve cross related entries in each of the Directories.
SeaDataNet from data discovery to unified access to analysis
Common Data Index (CDI) Data Discovery and Access Service • CDI is a fine-grained index to individual data measurements. The CDI Service provides users a highly detailed insight and unified access to the large volumes of marine and oceanographic data sets • An intelligent middle tier connection is configured between the SeaDataNet portal and the local data management systems at each of the data centres. • A shopping basket allows users to submit a shopping request for multiple data providers in one go and to follow its processing by each of the providers via an online transaction register.
Common Data Index (CDI) Data Discovery and Access Service Distributed Data Management
Data Policy and implementation • Metadata • free and open access, no registration required • each data centre is obliged to provide the meta-data in standardized format to populate the catalogue services • Data and products services • the general case is free and open access • however users must register once in order to get a personal login - password • Web form to provide necessary information • User agrees with “SeaDataNet User Licence” • After processing, login/password sent by email (email check) • Licence is part of the SeaDataNet Data Policy, that is intended to be fully compatible with the Directive of the European Parliament and of the Council on public access to environmental information, the INSPIRE Directive, IOC, ICES, WMO, GCOS, GEOSS and CLIVAR data principles. • SeaDataNet Data Policy is an overarching policy, respecting any local policy • Each user gets a SeaDataNet Role; while each CDI record has a ‘Data access restriction’ label. The combination of role and access label determines ‘access OK’, ‘access to be discussed’ or ‘access denied’
SeaDataNet CDI – Data retrieval and downloading Check Status In RSM Search Request Confirmed Include in Basket Results Ready at DC x Download Shopping list Data SDN format Submit + Authentication
SeaDataNet CDI Quick Search interface • Drilling down and very intuitive for any user
SeaDataNet CDI Extended Search interface • Searching for nutrients
SeaDataNet CDI Extended Search interface • Details of a selected CDI record
SeaDataNet CDI Search interface • Searching for nutrients – results map zoom in and also with Blue Marble
SeaDataNet CDI Extended Search interface • Shop for nutrient data around Danmark
SeaDataNet CDI Search interface • Shopping list for nutrient data around Danmark Submit
CDI extra functionalities • Numerous options for searching and drilling down of search results • CDI XML format can handle points, detailed tracks and polygons as GML • CDI Summary function for search results with statistics and drill-down function • CDI Mapping supports OGC WMS services with option to add external WMS layers and vice versa to serve CDI as WMS – WFS layer to other portals • A query and GIS layer configuration can be stored by users as a favourite and can be forwarded to other users as a URL • Extra attribute included for linking to (pre)viewing services
SeaDataNet CDI Search interface • Searching for nutrients – results Summary
SeaDataNet CDI Search interface • Searching for nutrients – results Summary – Data centres - clickable
SeaDataNet CDI Search interface • Add OGC WMS layers – EMODNet Hydrography
SeaDataNet CDI Search interface • OGC WMS layers added – EMODNet Hydrography
OGC WMS – WFS support • CDI entries from the CDI portal are available as a series of OGC WMS and WFS services, e.g. for inclusion in the EMODNet portals.
CDI - INSPIRE compliance • CS-W service on top of the CDI portal to deliver CDI XML entries in the INSPIRE compliant ISO 19139 format (at present aggregated by discipline and by data centre)
SeaDataNet Viewing services • Ocean Data View (ODV) • General data analysis and visualization software • >10,000 registered users • ODV software upgraded to ODV4 for: • Extending ODV‘s graphical display capabilities and interactive controls for automatic and visual quality control and data quality flagging • Seamless connection to SeaDataNet output: SDN ODV4 data formats, quality flag scale • Integration of DIVA gridding software • Important tool for the SeaDataNet regional Data Products • DIVAsoftware (Data-Interpolating Variational Analysis) allows to spatially interpolate observations on a regular grid in an optimal way. The analysis is performed on a finite element grid allowing for a spatial variable resolution and a good representation of the coastline and isobaths.
Regional Products services Climatologies and interpolated data products for the Arctic seas, Atlantic Ocean,Baltic sea, Mediteranean sea and the Black Sea regions. Statistical output in NetCDF files, that are visible via WMS through Ocean Browser
Cross sector involvement • SeaDataNet cooperates and provides services for many other European and international groups and projects, contributing and safeguarding good data management. Examples: • EuroGOOS, maintaining the European Directory of Ocean observing systems (EDIOS) and quality controlling and archiving long-term data series. See www.edios.org • GMES Marine Core Services for streamlining the provision of long-term archives for optimising marine forecast services; there is an MoU formulating the cooperation between MyOcean and SeaDataNet • POGO, collecting and providing information on ocean-going research vessels, and their operators, planned and completed cruises. See www.pogo-oceancruises.org • Services for the international oil & gas industry for metocean data (SIMORC). See www.simorc.org
SeaDataNet Expansion • SeaDataNet standards and tools are adopted by other EU-funded projects: • Up-Grade Black Sea SCENE project, (2009 – 2011) involving 6 NODCs and 35 other data holding institutes from the 6 Black Sea countries, to provide metadata and data access and to strengthen their national NODC networks. EU-funding 3.4 Million Euro. • CASPINFO project, (2009 – 2011) involving 12 institutes and private industry from the Caspian Sea region, to provide metadata and data access and to build their capacity for data management and user services. EU-funding 0.8 Million Euro. • Geo-Seas project, (2009-2012) involving 24 geological and geophysical data centres from 16 European countries (EuroGeoSurveys), to provide metadata and data access. EU-funding 4.9 Million Euro. • EuroFleets project, (2009-2012) involving 24 research institutes and data centres from 17 European countries, to optimise the management of research vessels and to streamline the flow of data from the research cruises to the data centre infrastructure. EU-funding 7.2 Million Euro. • In each project several SeaDataNet partners are participating for transfer of knowledge & expertise, to secure interoperability and to achieve crossfertilisation leading to upgrading of the SeaDataNet standards and tools as well as enriching of the services.
SeaDataNet and EMODnet • EU has adopted a new Marine Strategy Framework Directive (MSFD) which includes a requirement for an overarching European Marine Observation and Data Network (EMODnet) • EMODnet can be considered as a network of existing and developing European observation systems, linked by a data management structure covering all European coastal waters, shelf seas and surrounding ocean basins’.
SeaDataNet and EMODnet • SeaDataNet has qualified itself as leading infrastructure for the EMODnet data management component and is undertaking several preparatory EMODnet projects: • Chemistry – focus on the groups of chemicals required for monitoring the MSFD for establishing Good Environmental Status – cooperation with EEA, ICES and Regional Conventions • Hydrography – to produce and deliver a higher resolution digital bathymetry for European seas • Physics – together with MyOcean and EuroGOOS Regions to give wide access to operational metocean data, real-time and archives • The Geology lot is producing harmonised seabed maps (WMS). The SeaDataNet infrastructure is adopted via the Geo-Seas project to give overview and access to the underlying geological and geophysical data sets. • SeaDataNet is data partner in the Biology lot and a closer technical relation is foreseen in the SeaDataNet II project.
SeaDataNet and EMODnet • The EMODnet pilots and associated research projects stimulate a wider uptake of SeaDataNet standards and contribute to enriching the standards and services • SeaDataNet is a project with a fixed partnership. The EMODnet pilots are data driven and bring on board many more data providers that adopt the standards to connect to the infrastructure and to populate the data resources. • To illustrate this: SeaDataNet comprises 40 Data Centres, but already 60 Data Centres are now connected and we are making good progress towards ca 90 connected Data Centres • EMODnet is management and policy driven which also provides an excellent opportunity for SeaDataNet to upgrade its status from project to sustained operational infrastructure.
CDI Service as common service www.emodnet-chemistry.eu www.blackseascene.net www.geo-seas.eu SeaDataNet CDI Service www.emodnet-hydrography.eu (www.emodnet-physics.eu) www.caspinfo.net
CDI – Data Coverage CDI coverage at end May 2011: > 1.000.000 CDI entries from 29 countries and >60 data centres for physics, chemistry, geology, geophysics, bathymetry and biology; connecting another 30 data centres and increase in CDIs is well underway
SeaDataNet II proposal – Recently accepted There are many challenges and recent innovations which will be taken up in SeaDataNet II that will run from 1st October 2011 – 30th September 2015 with a funding budget of 6 Million Euro. A number of objectives: • Achieving full INSPIRE compliance and contributing to the INSPIRE process for developing implementing rules for oceanography • Achieving data access and data products services that meet requirements of end-users and intermediate user communities, such as GMES Marine Core Service, establishing SeaDataNet as the core data management component of the EMODNet infrastructure, and contributing on behalf of Europe to global portal initiatives, such as the IOC-IODE - Ocean Data Portal (ODP), and GEOSS. • Achieving an improved capability for handling also marine biological data and interoperability with the emerging biodiversity data infrastructure in Europe
SeaDataNet II project – selected technical aims • Establishing machine-to-machine interfacing next to present user portal for technically serving specific user communities, such as GMES Marine Core Service and WISE-Marine (EEA) • Developing online visualisation (previewing, viewing) services • Extension of the CDI metadata format with Observations & Measurements module and SensorML module, where applicable for specific types of data, e.g. for operational oceanography • Formulation of NetCDF core data model including SeaDataNet metadata header in cooperation with UniData • Tackling the duplicates problem
Conclusions • SeaDataNet is a strong partnership, well established in the Member States and in the ocean and marine community • SeaDataNet operates and further develops a data management infrastructure that is adopted as core data management component in the EMODnet pilots and also adopted by many research projects and monitoring initiatives • SeaDataNet has established and is further developing marine data management standards that are adopted on a wide scale • SeaDataNet has established and is further developing interoperability solutions, such as machine-to-machine interfaces, for serving specific user communities • Associated projects and EMODnet pilots stimulate expansion of the number of connected data centres and increase of data population, which is very relevant for MSFD and WISE-Marine.
EMODnet opportunities • Additional funding (22.5 MEuro) is being arranged for 2012 – 2014 for more data pilots and extending and operating existing pilots. • Moreover EMODnet has the promise of a substantial and structural funding from 2014 onwards for the data management component and the observation systems. • Already the preparatory actions are accelerating the convergence between different data sectors (geology, hydrography, chemistry, ….). There is an increasing willingness to harmonise, to become interoperable and to enable sharing of data in sectors and between sectors. • These opportunities and the SeaDataNet – EMODnet momentum should be used by DG Environment and EEA.
MSFD brings EMODnet and WISE-Marine together Policy relevant data Marine Core Service data Country data Reference layers (maps) Research data EuropeanMarine Observation Data NETwork MSFD WISE-Marine