310 likes | 479 Views
SeaDataNet mid-term review – Bologna – September 2008, 19. SeaDataNet Pan-European Infrastructure for Ocean and Marine Data Management. Gilbert Maudire and the SeaDataNet Consortium. Data is of vital importance for marine research and support to public decision.
E N D
SeaDataNet mid-term review – Bologna – September 2008, 19 SeaDataNetPan-European Infrastructure for Ocean and Marine Data Management Gilbert Maudire and the SeaDataNet Consortium
SeaDataNet mid-term review – Bologna – September 2008, 19 Data is of vital importance for marine researchand support to public decision • Marine observation and data sets status • observation conducted by more than 1100 active scientific laboratories in the countries of partners, using various sensors and platforms • not always quality controlled using comparable procedures • not always easily accessible • Necessity of large background work in order to : • secure them in long term archives • document available observations and data sets (metadata) • improve coherence of data sets (references, quality, redundancy) • make data sets accessible easily (formats, on line distribution)
SeaDataNet mid-term review – Bologna – September 2008, 19 What is SeaDataNet? • EU Sixth Framework Programme • Integrated Research Infrastructure Initiative (I3) • 1 April 2006 to 31 March 2011 (5 years) • Objectives • To network existing oceanographic data centres already nationally funded • To develop an efficient distributed pan-European marine data management Infrastructure (a “unique Virtual Data Centre”) • To provide on-line access to distributed databases of standardised quality by using adapted communication & information technology • This project integrates and develops several EU past initiatives and international cooperation, in particular Medar/Medatlas, Edmed, Edios, Sea-Search etc.
SeaDataNet mid-term review – Bologna – September 2008, 19 49 Partners • National Oceanographic Data Centres • divisions of major national marine research institutes in 35 countries • experienced as leaders of data Management of many scientific projects and of concerted actions (e.g. MTPII-MATER, CANIGO, OMEX, EDIOS, MERSEA, MEDAR/MEDATLAS, SEA-SEARCH) • 3 satellite data centres • 3 scientific modelling centres • 2 SMEs experts in software development • 3 International organisations with strong regional interest/cooperation
SeaDataNet mid-term review – Bologna – September 2008, 19 SeaDataNet managed data • Primary focus on water column parameters • Physical • Chemical • Bio-chemical (non species dependant) • Projects will be proposed for other domains by involved communities in strong relationships with SeaDataNet Several SeaDataNet partners are already involved in those domains • Geophysical and geological data • Biology (species dependant)
SeaDataNet mid-term review – Bologna – September 2008, 19 SeaDataNet services • Online services : web portal and web services • Discovery services • Metadata directories • Common vocabularies : parameters, units, … • Delivery services • Security services : Authentication, Authorization & Administration (AAA) • Data download and access including reformatting in standardized formats • Viewing services • Web map services, data visualization. • Offline services : « back office » services • Safeguarding within the national repositories (NODCs) • Quality Control and desktop viewing services • ODV analysis software • Product generation adapted to each regions • Monitoring and statistics : usage & system performance
SeaDataNet mid-term review – Bologna – September 2008, 19 Infrastructure versions & planning • Version 0 – 2006-2007 Continuation and maintenance of existing Sea-Search system : the data access needs several different requests to each data centres and the data sets are delivered in different formats • Operated at the start of the project • Version 1 – 2008-2010 Setup of the integrated online data services to users : networking of 10 “interoperable” data centres of the Technical Task Team unique request to the interconnected data centres and the data sets are delivered with a unique format • Presently under test and progressive integration of 10 data centres during 2008 • Version 2 – 2010-2011 Improved version of the system (ease of use, data visualisation, data access)and by step, all data centres connected for providing online trans-national access to the data, metadata and products • Operational in 2010
SeaDataNet mid-term review – Bologna – September 2008, 19 Discovery services All catalogues(European metadata directories) • are compliant with ISO19115 standard family for spatial metadata • make use of common vocabularies and references • are accessible and linked through web services • ease data access using various selection criterions adapted to different users (scientists, observatory and fleet managers, …)
SeaDataNet mid-term review – Bologna – September 2008, 19 • Common data index : gate to data access • online searchable directory where?, when?, what?, … • tool for managing user’s“Shopping Basket”, that is : • To prepare a list of distributed datasetsmatching user’s request • To ask concerned data centres to serve the user, • To display the status of user’s request • To route the requested data from the distributed data centres to the user
SeaDataNet mid-term review – Bologna – September 2008, 19 SeaDataNet data policy and data licence inspired by INSPIRE Directive to facilitate the data flow takes into account the local rules and the SeaDataNet users needs • Metadata • free and open access, no registration required • each data centre should provide the meta-data in standardized format to populate the catalogue services • Data and products • visualisation freely available • data download, in general, free for academic purposes with mandatory registration • a “SeaDataNet role” (partner, academic, commercial etc.) is attributed to individual user using the Authentication, Authorization and Administration (AAA) Service • Each NODC attributes the roles to the users of its of country • Out of the partnership, the roles are assigned by SeaDataNet user-desk • a common data licence has been adopted and users must agree on when registering • each data centre node delivers data according to the user’s role and its local regulation • each data centre should provide the data sets necessary to develop the common products
SeaDataNet mid-term review – Bologna – September 2008, 19 QC procedures and tools • SeaDataNet protocol : • Coherent automatic and visual checks are performed on the data by each NODC • A unique quality flag scale has been adopted • The automatic checks and the visualization could be performed using either by Ocean Data View (ODV) or existing partner software (if the same protocol and checks are applied) Example:Using derivatives for spike & discontinuity detection in ODV
SeaDataNet mid-term review – Bologna – September 2008, 19 Data products • SeaDataNet develops added-value products at regional scales: • Mediterranean Sea • Black Sea • Baltic Sea • Arctic Seas • Atlantic and global ocean The preparation of these products allows thecheckingof the compatibility of the QCs performedon the data in each NODC. • First products => gridded climatologies and trends • The comparability of the different regional products is assessed by using a new version of DIVA (Data Inverse Variational Analysis) developed (University of Liege)
SeaDataNet mid-term review – Bologna – September 2008, 19 • Training program main objectives • to ensure that management procedures are coherent within the consortium • to transfer expertise and to train IT experts of the SeaDataNet data centers in the installation and operation of the SeaDataNet technical components Capacity Building is a main issue
SeaDataNet mid-term review – Bologna – September 2008, 19 www.seadatanet.org Please contact : sdn-userdesk@seadatanet.org
SeaDataNet mid-term review – Bologna – September 2008, 19 SeaDataNetProject management
SeaDataNet mid-term review – Bologna – September 2008, 19 Role of partners (1/4) • Coordination • Ifremer : project coordinator • Maris : technical coordinator animator of the Technical Task Group with the help of : • NERC/ BODC : standard development • AWI : data quality control and desktop viewing services • University of Liege : scientific coordinator in charge of the regional products development with the help of : • JRC : Product harmonization • INGV : coordinator of Mediterranean products • METU : coordinator of Black Sea products • NERI : coordinator of Baltic products • IMR & VLIZ : coordinators of Artic & North Seas products • Ifremer : coordinators of Atlantic & Global Ocean products
SeaDataNet mid-term review – Bologna – September 2008, 19 Role of partners (2/4) • Coordination (cont.) • HCMR : Coordinator of the networking activities i.e. metadata and data management, system monitoring with the help for coordinating the distributed network of • Maris : management of Common Data Index (CDI) and of Directories of Marine Projects (EDMERP) and Marine Organizations (EDMO) • NERC/ BODC : management of Directories of Marine Environmental Datasets (EDMED) and of Ocean Observing System (EDIOS) • BSH : management of Cruise summary reports • Ifremer : Management of SeaDataNet users
SeaDataNet mid-term review – Bologna – September 2008, 19 Role of partners (3/4) • Technical task team (11 partners : Western Europe + Russia) • Technical specifications, • Check of the defined standards • Integration and system setup, test and validation of technical components • Running of System V1 • Capacity building and training • UNESCO/ IOC Project office with the help of Technical Task Team • Information and communication • ENEA
SeaDataNet mid-term review – Bologna – September 2008, 19 Role of partners (4/4) • All partners which are mostly NODC’s (except international organizations) have to • generate metadata : organizations, projects, cruises, observing systems, data sets • manage data from their country : quality, format, … according to SeaDataNet agreed standards • provide access to their data repository, automatically using SeaDataNet system or manually as a first stage • manage users of their respective countries by providing information and help
SeaDataNet mid-term review – Bologna – September 2008, 19 External relationships / Collaborations • Advisory board • MEDS Canada(represents IODE/WMO JCOMM, GTSPP and Argo international programs) • US – NODC (represents North America initiatives like DMAC) • EuroGOOS(represents Operational Oceanography community) • Standard summit (last January on behalf of COI) • North America, Europe, Japan • “Memorandum of understanding” with other consortiums • Unidata, Mersea-MyOcean (GMES), EuroBis-Marbef (FP6), GENESI (FP7) • International conference on Marine Data and Information Systems (IMDIS) • April 2008, more than 200 participants from about 50 countries.
SeaDataNet mid-term review – Bologna – September 2008, 19 Project management tools • SeaDataNet web sitewww.seadatanet.org • Distributed management, based on a Content Management system (EZPublish) • General information : Description of Work, Meetings, Conferences, Education (several languages) • Project services (discovery, data access, …) • Finalised documents (applicable version of standards) and software tools • Mailing lists • Based on Sympa software • One list per group (technical, …) owned by the chair person • + user desk (SeaDataNet users point of contact) • Document management • Based on BSCW software (Basic Support for Cooperative Work - Germany) • Cooperative work on documents (non finalised versions) and software • Software management (versionning tool) • Based on GForge
SeaDataNet mid-term review – Bologna – September 2008, 19 System monitoring focused on system “availability” • Availability of SeaDataNet user services • availability of web pages and web services general information, metadata browsing (EDMED, CSR, CDI), data selection users’ interfaces, web services • availability of web data access servicesdata centre download services • Service “available” if monitoring system get an answer • Search of keywords within web pages, • Web services can be reached : WSDL returned • Monitoring • done by IFREMER (coordinator) and HCMRin order to avoid biases due to network troubles • Using NAGIOS Open Source software • Polling of services (web pages, web services)
SeaDataNet mid-term review – Bologna – September 2008, 19 Polling of system components
SeaDataNet mid-term review – Bologna – September 2008, 19 Monitoring board • Example of one-month report for Ifremer http access Electrical failure during 6 hours
SeaDataNet mid-term review – Bologna – September 2008, 19 SeaDataNetAdministrative and financial aspects
SeaDataNet mid-term review – Bologna – September 2008, 19 Known issues • IOC Project Office partner • In charge of training courses and capacity building • Report at 0 up to now even if work done (3 training courses organised yet)no agreement on US $ / € change rate between UNESCO and EC) • IOC is an « Additional Cost partner »not well adapted to training course organization (additional personnel) • Transnational activities report • Partners with very small web site and/or very little audience • Audience measurment : KiloPageViews unit difficult to report accurately • 80% of partners report less than expected for TA
SeaDataNet mid-term review – Bologna – September 2008, 19 Transnational Activities amendment • Change reporting unit from KiloPageViews to KiloHits • To take into account all hits on online services :web pages (for any usages), web services, ftp servers, … • But, cost / unit must decrease (Project Officer comment) • However, most audiences will remain far less than expected • Cost are not linearly proportional to the audience« Entry costs » to set up a web site funds transfer to JRA (web site maintenance) • Who? How much? • All must remain in the same budget envelope
SeaDataNet mid-term review – Bologna – September 2008, 19 SeaDataNetPlanning
SeaDataNet mid-term review – Bologna – September 2008, 19 Infrastructure versions & planning • Version 0 – 2006-2007 Continuation and maintenance of existing Sea-Search system : the data access needs several different requests to each data centres and the data sets are delivered in different formats • Operated at the start of the project • Version 1 – 2008-2010 Setup of the integrated online data services to users : networking of 10 “interoperable” data centres of the Technical Task Team unique request to the interconnected data centres and the data sets are delivered with a unique format • Presently under test and progressive integration of 10 data centres during 2008 • Version 2 – 2010-2011 Improved version of the system (ease of use, data visualisation, data access)and by step, all data centres connected for providing online trans-national access to the data, metadata and products • Operational by 2010
SeaDataNet mid-term review – Bologna – September 2008, 19 Project planning and budget • High level of effort during the first three years(80% of the budget) • Common standard adoption, set up and training, • Development / Adaptation of common tools • System specifications, development and setup (V1 and then V2) • Development of first release of regional products and initial harmonization • Lower level of effort during the last two years • Continuous data management • Exploitation of the system, system maintenance and upgrade • Upgrade of regional products
SeaDataNet mid-term review – Bologna – September 2008, 19 More information about budget • Funds breakdown / year / partner after second report • Funds breakdown : https://www.ifremer.fr/bscw/bscw.cgi/0/110042 • Transnational access audience • Evolution for TA- Seadatanet.xls