1 / 25

EMODnet Chemistry 2 Various actions By Dick M.A. Schaap – Technical Coordinator

EMODnet Chemistry 2 Various actions By Dick M.A. Schaap – Technical Coordinator Rome – Italy, 20 – 21 January 2014, Steering Committee. Indicators for reporting to Secretariate.

angus
Download Presentation

EMODnet Chemistry 2 Various actions By Dick M.A. Schaap – Technical Coordinator

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EMODnet Chemistry 2 Various actions By Dick M.A. Schaap – Technical Coordinator Rome – Italy, 20 – 21 January 2014, Steering Committee

  2. Indicators for reporting to Secretariate • 1) Volume of data made available through the portal: All gathered data are included in the CDI Data Discovery and Access service => the number of CDIs and the increase over time gives an excellent indicator to follow progress. • 2) Organisations supplying each type of data: => related to CDIs we can provide the numbers divided over data centres and their countries. And thus the increase over time • 3) Organisations that have been approached to supply data with no result: This has to be treated with care, because part of the remit of the projects is to identify potential additional data providers and data sets and then to encourage and/or negotiate them to make these part of the data discovery and access services in EMODNet. Also this process takes time. => we propose to report this as a management report and on an annual basis.

  3. Indicators for reporting to Secretariate • 4) Volume of each data type downloaded from each portal: Users undertake shopping for data for which they need to be registered and login. All requests and actions for downloading of data sets are registered in the SeaDataNet Request Status Manager (RSM) which allows for reporting. The RSM registers which CDIs, which users, which connected data centres and at what time. It also reports these per portal used (Chemistry, Bathymetry, Physics, ..) However analysing these in qualitative respect is beyond our scope of work and can be quite laborsome. => we propose a more concise report from the RSM: how many CDIs were requested by how many users from different data centres over a given time period from a given portal. This will give already a good indication of data downloading. In stead of reporting these on a 2-monthly basis we rather do this on a 6-monthly basis. In addition there is browsing and downloading of data products (interpolated sea region maps for specific chemical substances) => it is possible to report a table of downloaded data products over a given time period. In stead of reporting these on a 2-monthly basis we rather do this on a 6-monthly basis. • 5) Organisations that have downloaded each data type: => For CDI from RSM we can provide per portal the names of user organisations that have requested and downloaded so many data sets. Qualitative analysis of what kind of data is a lot of effort and out of scope. The report is more quantitative. Like 4) this is proposed on a 6-monthly basis.  However we must be careful with privacy of individual users!! => For data products no tracking takes place of users, because viewing and downloading of data products is public, free and anonymus.

  4. Indicators for reporting to Secretariate • 6) Using user statistics to determine the main pages utilised and to identify data products being used: We make use of web statistics, e.g. AW Stats, to get info on the visits to the web portals and their different sections and services. Giving a summary table from these statistics services is easy, but analysing the statistics more in depth is complex and out of the scope of the planned work for the portals. => we can ararnge that the Secretariate gets online access to these statistics reporting systems so that it can make its own analyses. • 7) List of what the downloaded data has been used for: The portals do not have insight in the purpose of downloading data sets by users. => it is proposed that the Secretariate will make use of the reported users (see 5) to do some following up towards specific users, asking more about their purpose, satisfaction, etc.. • 8) List of organisations that have downloaded data from more than one portal: This requires concertation between portals. => It is proposed that the Secretariate will make use of the reported users (see 5) to do this kind of cross analysis. • 9) Interoperability of data of different types and from different portals: => It is proposed to discuss this in the SC and as part of the experiences of the Use Cases and the Checkpoint Studies.

  5. Indicators for reporting to Secretariate • 10) Monitoring levell of interaction with member states national processes for data stewardship: As discussed at the SC meeting this will not be easy. => Where it is possible and appropriate it is proposed to report this on an annual basis as part of the Annual Reports. • 11) Management Budget Overview: As discussed and agreed at the SC this will NOT be supplied by the projects because the work is undertaken on a lump sum basis.  DG MARE will consider its own input and the time / money spent on the Secretariate as Overhead, rest as Research and Development. • 12) EMODNet Citations: => It is proposed that the project reports on citations (articles in publications, presentations at conferences etc) on an annual basis as part of the Annual Reports.

  6. EMODnet Chemistry portal – present set-up

  7. Working with a central buffer to support products generation and additional visualisation • SeaDataNet II is making good progress with automatic harvesting using a robot user in the CDI system in order to maintain internal data buffers for specific purposes • Robot harvester is operational and has been applied for automatically harvesting ca 1.000.000 T&S data sets as part of the joint data climatology product development SeaDataNet – MyOcean • Robot harvester can be configured for specific data buffer profiles and in agreement (SLA’s) with data providers AND specific data user communities (such as MyOcean, SeaDataNet regional dataproduct groups, EMODNet Chemistry regional product groups, WG DIKE); work is ongoing for extending the functionality of the RSM service for this purpose

  8. Working with a central buffer to support products generation and additional visualisation • the central buffer system with set of database views will be maintained automatically by the RSM - robot user system, following new and updated CDI submissions • the access to these buffers of aggregate data sets can also be virtual for specific Workflow Processes (WPS) in which case only results and graphics of analyses are made available, not the individual data • further work ongoing for • adding a controlled access to the buffer data sets for specific purposes, using the RSM administration, • exploring options for hosting the buffer system in the cloud • plus developing additional subsetting and visualisation services • See also SeaDataNet II Deliverables D5.6 and D8.7

  9. Planned architecture extension Oceanotron OpenEarth DIVA ODV Advanced access and viewing services CDI User Interfaces CDI Robot harvester Dynamic Maintenance Specific data buffers Agreed Settings

  10. Converting buffer data sets to validated aggregated data sets • the central buffer system will contain ‘raw’ data sets for specific parameters and as harvested from the distributed data centres => further action needed for making the collection more homogeneous and validated => aggregated data sets • this can be done by using ODV software and specific expertise per region and chemical substance • use will also be made of the new P35 Vocabulary for aggregating P01 terms. The P35 pilot is already up for one P35 term: http://seadatanet.maris2.nl/v_bodc_vocab_v2/welcome.asp => further populating of the P35 Vocabulary list requires contributions from domain experts who can judge which P01 terms scientifically can be grouped under specific P35 terms. BODC will ingest entries following experts proposal.

  11. EMODnet Chemistry extension DIVA + OceanBrowser service Oceanotron & OpenEarth services QCd data buffers Regional experts ODV QC + aggregation Specific data buffers CDI Robot Harvester

  12. EMODnet Chemistry planned output services • Next to already existing ODV -> Diva -> OceanBrowser service for interpolated maps • Use the robot harvester to build a central buffer of data sets with regular updating intervals • Extract relevant 'raw' data files for EMODNet Chemistry from the central buffer • QA/QC and aggregate these into a homogeneous ODV collection which sits in ODV • Evaluate the advances of Oceantron (within SeaDataNet) for having access to the ODV Collection, using the ODV API and appropriate Oceanotron plug-in, and making them available for OGC SWE and WMS services • WPS processes will take the OGC compliant output (possibly from Oceantron) as their input and generate on the fly added value graphics such as station time series, concentration plots for a given time and space window, etc • These new WPS products will be described in the SEXTANT Catalogue service • The new WPS products will be served to users via the OceanBrowser viewing service

  13. Draft specifications for cloud hosting (prelimininary) • Sufficient storage • Pref. accessable as an external harddisk; otherwise minimally by FTP • Machine to machine interfacing, such as automatic login etc. • Reading of statistics • Security • Using SeaDataNet Caslogin must be possible • Ip-number security desirable for setting up IP to IP connections • Possible support for OpenID access • How can we configure and control rights of the cloud towards accessing MARIS systems • How safe is the system on which the cloud operates • Possibility by MARIS to develop, include and run own software, such as SOAP (machine to machine) • Support for platforms (windows en unix); for MARIS windows is 1st requirement • Possible to develop and run Webservices (SOAP) (windows, php, ..) • Possible to make and operate service for preparing and downloading of zipfiles. • Possible to make and operate internet interface. • Making use of storage and of database, eg mysql

  14. Filter P02 – P01 for EMODNet Chemistry scope • EMODNet Chemistry 2 scope is enlarged compared to previous Chemistry 1 • MARIS uses the P02 scope in the CDI Data Discovery and Access service for subsetting the EMODNet Chemistry view from the overall SeaDataNet CDI database • The P02 – P01 tables are used for the data providers to support their data gathering and mapping for CDI and ODV preparation • MARIS has asked BODC, AWI and ICES for advise • BODC has prepared a first draft; MARIS has submitted the previous list • Needs finalisation (and possible extension of Vocabs P02 – P01 ?)

  15. CDI Data Discovery and Access Service – Sea regions C19

  16. CDI Data Discovery and Access Service – Sea regions C19 – polygons for marking up all CDI entries with C19 terms using spatial queries + extra tags via C19 BT relations

  17. CDI Data Discovery and Access Service – Sea regions Example – Search by North Sea in CDI User Interface

  18. Implication of data central buffer for product generation and new coastal visualization tools as discussed at the TWG • MARIS can help regional groups for gathering and collating by Robot all relevant ODV data sets and associated CDIs per region => provision of raw data sets by DVD; later in the project by downloading from the cloud OR WPS • Action TWG-10: coastal issue: developing an extra spatial query functionality in the CDI User Interface for searching based on the distance along the coastline. Also an external WMS layer with geographical features (e.g. towns, borders, river outlets) by means of OpenStreetMap will be added by MARIS (March 2014) • Action TWG-11: coastal issue: developing the visualization along (or near) the coastline of a Chemistry data product in OceanBrowser. One axis of such visualizations will correspond to the distance along the coastline from a given starting point. Currently vertical sections can be extracted along a user-specified path. Data extraction along the coastline will be simplified for providing a list of predetermined sections following the coastline. The display of the along-coast figures will be optimized to take into account the specificities of these. By ULg (March 2014)

  19. Implication of data central buffer for product generation and new coastal visualization tools as discussed at the TWG • Plus extra products to be produced by OpenEarth via WPS into OceanBrowser: • visualise a time evolution of a selected group of measurements • show concentration plots for a given time and space window and also along the coast • show inflows from rivers of nutrients. The user should be able to select a section of coast, a country or a region (NUTS3) and obtain time series of inflows of parameters expressed as mass or moles per unit time per river (or section of coast)

  20. TWG – activities for EMODNet Chemistry portal • CDI format and service upgraded to ISO 19139 and ISO compliance • CDI User Interface extended with • search for P08->P03->P02 • Search for Sea Regions (C19) • Search for time series • CDI Details extended with EDMED, CSR, • Display of time series in Results • Possible extra: • Geocoding and search for MSFD regions • Coastal searching • Open Street View as one of the basis map layers • Links between Central Portal and EMODNet Chemistry: • Use of WMS – WFS services • SEXTANT Catalogue of products => subset to be shared in Central • Dedicated use cases => specific customised queries / services?

  21. ACTIONS from TWG-1 • Action TWG-1: a pilot will be set-up for exposing the P01 semantic model based upon JSON (XML) provided as service by BODC and upgraded client interface provided by MARIS including functions for searching and filtering on separate fields and searching across fields (end 2013 - early 2014 ) • Action TWG-2: the set-up of the pilot and its progress to date will be presented by BODC and MARIS at the coming 2nd ODIP Workshop (December 2013) for discussion with international experts and also for further brainstorming on possible RDF application • Action TWG-3: the P35 Vocabulary list will be set-up by BODC including possibility to relate to P01 terms and MARIS will include it in the SeaDataNet client interface (asap) • Action TWG-4: populating the P35 Vocabulary list requires contributions from domain experts who can judge which P01 terms scientifically can be grouped under specific P35 terms. BODC will ingest entries following experts proposal. • Action TWG-5: incorporating the P35 Vocabulary list into the ODV aggregation software will be developed by AWI (spring 2014) • Action TWG-6: incorporating the P35 Vocabulary list into the Diva Products Discovery service (via SEXTANT Catalogue) will be analysed by ULG and IFREMER (end 2013)

  22. Semantic model for P01 • Proposed fields to expose • Measurement (including classifier) • Substance (including classifier, alternative name) • Measurement Matrix Relationship • Matrix • Matrix Subcomponent • Taxon • Organism Name • Organism Specifics • Technique • Example:XMETAL04 • Measurement: “Concentration” • Substance: “cadmium {Cd}” • Measurement Matrix Relationship: “per unit wet weight” • Matrix: “biota” • Matrix Subcomponent: “liver” • Taxon: “Platichthys flesus (ITIS: 172894: WoRMS 127141)” • Organism Name: “Platichthys flesus” • Organism Specifics: “Sex: male Size: length 226-250mm” • Technique: “inductively-coupled plasma mass spectrometry”

  23. ACTIONS from TWG-1 • Action TWG-7: formulating a plan incl proposed format (JSON?) for expanding the new P01 discovery client interface with an online form procedure for requesting and first level validating new P01 codes as combinations of fields by BODC and MARIS for discussion at ODIP 2nd Workshop (end November 2013) • Action TWG-8: implementing the online form procedure for requesting and first level validating new P01 codes as combinations of fields by BODC and MARIS (April 2014) • Action TWG-9: exploring extending the CDI discovery interface with (smart) search on chemical substances by BODC, IFREMER and MARIS (period mid 2014 - mid 2015) • Action TWG-10: coastal issue: developing an extra spatial query functionality in the CDI User Interface for searching based on the distance along the coastline. Also an external WMS layer with geographical features (e.g. towns, borders, river outlets) by means of OpenStreetMap will be added by MARIS (March 2014) • Action TWG-11: coastal issue: developing the visualization along (or near) the coastline of a Chemistry data product in OceanBrowser. One axis of such visualizations will correspond to the distance along the coastline from a given starting point. Currently vertical sections can be extracted along a user-specified path. Data extraction along the coastline will be simplified for providing a list of predetermined sections following the coastline. The display of the along-coast figures will be optimized to take into account the specificities of these. By ULg (March 2014)

  24. ACTIONS from TWG-1 • Action TWG-12: communication between IFREMER and ULg about the upgraded SEXTANT metadata format for upgrading the Diva2XML application and how to harvest Diva XML entries (asap) • Action TWG-13: communication between IFREMER and ULg about the alignment of the present content because at the moment the two sets of maps are different (asap) • Action TWG-14: planning and implementing by IFREMER and ULg mutually linking of SEXTANT and OceanBrowser; SEXTANT entries must give direct access to viewing of the selected product in OceanBrowser; Product Index in OceanBrowser must be fed by SEXTANT and include a SEXTANT link per product for detailed description (plan end November 2013; implementation March 2014) • Action TWG-15: optimising the SEXTANT discovery interface as a dedicated EMODNet Chemistry service, possibly including P35 and sea regions as extra search criteria and considering OceanBrowser as viewer by IFREMER (March 2014) • Action TWG-16: ULg will send a few ODV problem files to IFREMER to check whether NEMO can read and repair these (asap) • Action TWG-17: ULg will organise a Diva Workshop on Corsica - France 4 - 8 November 2013. • Action TWG-18: a pilot chain of services from ODV to Oceanotron to WPS will be formulated, deployed and evaluated by AWI, IFREMER and Deltares (March 2014)

  25. www.emodnet-chemistry.eu

More Related