1 / 23

SeaDataNet TTG meeting

SeaDataNet TTG meeting. Overview and potential use of E2EDM technology for SeaDataNet. Sergey Belov, Nick Mikhailov. Paris, 18-19 May. NODC, Russia. Scope. E2EDM provides following functionality: Wrapping local data systems Mapping local codes and dictionaries

kourtney-warren
Download Presentation

SeaDataNet TTG meeting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SeaDataNet TTG meeting Overview and potential use of E2EDM technology for SeaDataNet Sergey Belov, Nick Mikhailov Paris, 18-19 May NODC, Russia

  2. Scope E2EDM provides following functionality: • Wrapping local data systems • Mapping local codes and dictionaries • Data discovery, delivery and processing (export and data production from delivered data) • Integrated and secured online access to the distributed data E2EDM is integrating operational and delay-mode time scale data (oceanographic and marine meteorological data) from multiple data formats and multiple data providers in different geographic regions. NODC, Russia

  3. Basis • Concept model • Namespace • Metadata/data objects specifications • Data exchange protocol • Software components NODC, Russia NODC, Russia

  4. E2EDM concept model • Providing formalization, identification, description and interpretation for main model objects: • Data source • Resource • User (in particular cases – external application) • Model is based on the following structured semantic objects: • Element • Class - set of elements • Record - set of classes • These semantic objects defines: • Granularity and description for original and transferred data • Metadata description • Codes and dictionaries unification • Data and user policy NODC, Russia NODC, Russia

  5. E2EDM namespace • Defines metadata and data structures in terms of XML scheme. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic). • Global namespace – structures for the model objects • Conceptual (thematic domain) namespace – list of thematic elements dealing with grouping, e.g. some discipline and available for discovery and exchange • Both designed in XML Schema. • Global XML Schema (ISO 19115 + WMO Core Metadata + CDI) • Concept XML Schema(DiGIR) • XML Schemas are also used for validation and development issues (XML Schema -> UML -> software design) NODC, Russia NODC, Russia NODC, Russia

  6. E2EDM metadata objects specifications E2ELinkage E2EDataQuality E2EObjectID E2EDistributionInfo E2ERestrictions List of classes, defining Instruments and processing tools, projects, platforms, geographic coverage, data structure and granularity E2EVerticalExtent E2ETemporalExtent E2EObjectContact E2EObjectReference Information about temporal, vertical extent Metadata Information about local data source (linkage) Unique identification of metadata object Information about data quality Information about references and responsible party Information about data policy Information about data set More than 100elements, 27 classes, 7 metadata records and 4 data records NODC, Russia NODC, Russia NODC, Russia

  7. E2EDM data objects specifications Header (DataHeader) ListElement (list of elements) DataContainer «SequenceElement» «StructureElement» • E2EDM datarecords, based on NetCDF: • E2EDataPoint – for point data • E2EDataProfile – for profile data • E2EDataGrid – for grids • E2EDataObject – object files (http-based and local) Common transport NetCDF construction NODC, Russia NODC, Russia NODC, Russia

  8. E2EDM data objects specifications (II) Classes and data constructions relationship: <E2EObjectHierarchy> <objectHierarchyLevel>1</objectHierarchyLevel> <objectHierarchyList>objectSystemId, objectHierarchyId, countryId, organizationId, dateTime, Latitude, Longitude, platfType, platformId, platformName</objectHierarchyList> <objectHierarchyId>platformId</objectHierarchyId> <instanceId>platformId</instanceId> </E2EObjectHierarchy> <E2EObjectHierarchy> <objectHierarchyLevel>2</objectHierarchyLevel> <objectHierarchyList>DEPH, TEMP, PSAL</objectHierarchyList> <objectHierarchyId>dateTime, Latitude, Longitude</objectHierarchyId> <instanceId>dateTime</instanceId> </E2EObjectHierarchy> NODC, Russia NODC, Russia NODC, Russia

  9. E2EDM data exchange protocol Data exchange protocol is responsible for data requesting and data retrieving. Request message (XML) Response message (XML) Transport data file (NetCDF,v.2.2) NODC, Russia NODC, Russia NODC, Russia NODC, Russia

  10. E2EDM software components • E2EDM Data Provider (DP) – set of services, providing interface between local data systems and distributed system. Must be plugged on the local system. Services are using common library (E2EDM API). DP supports data bases, structured and object data files. • E2EDM Integration Server – set of services, providing security issues, resource catalogue maintenance and communication with DP (discovery, requesting and retrieving data) NODC, Russia

  11. E2EDM Architecture External layer Integration Server layer Communication layer Data Provider layer Local data system layer NODC, Russia

  12. Integration Server. Features • Metadata update & maintenance (automatic) • Resource catalogue management • Discovery services for data, metadata and common codes • Request/response communication service • Data sources monitoring • Downloading services (for obtaining data files, metadata, resource and instance descriptions synchronization) • Export tools (e.g. export in XML from E2EDM resource description into WMO Core Profile, to ASCII) • HTTP GET/POST (REST) and Web service interfaces • Security service • Schedule mechanisms • Administration tools (CRM, CMS) NODC, Russia

  13. Data Provider. Features • Data source/resources registry and metadata update • Codes and dictionaries mapping • Request handling • Access to the local data source (DBMS, files) • Data extraction based on search criteria • Transport data file generation • HTTP GET/POST (REST) and Web service interfaces • Security service • Schedule mechanisms • Administration tools (online maintenance for resource and instances) NODC, Russia

  14. General data exchange scheme User Interface layer (HTTP, Web-services) reply request Integration Server Transport data file Response message Request message Transport data file Data Provider Data Provider Data Provider NODC, Russia

  15. Current operational status of E2EDM VLIZ, Belgium • E2EDM was originated as one of the JCOMM/ETDMP pilot projects in 2003 MetOffice,UK IFREMER, France NODC, Russia NODC, Russia

  16. Pilot project data sources The following data sources are involved in the E2EDM prototype • Historical marine meteorological data • Historical ocean cruise data • Real-time GTS ocean (BATHY and TESAC) data • Real-time GTS ocean (TESAC/ARGO data) • Real-time GTS (SHIP) data • Monthly climatic fields of ocean parameters (imageries) • Ocean SST satellite data Geographic area – North Atlantic (test-area) NODC, Russia

  17. Pilot project data sources The following data sources are involved in the E2EDM prototype • Historical marine meteorological data • Historical ocean cruise data • Real-time GTS ocean (BATHY and TESAC) data • Real-time GTS ocean (TESAC/ARGO data) • Real-time GTS (SHIP) data • Monthly climatic fields of ocean parameters (imageries) • Ocean SST satellite data Geographic area – North Atlantic (test-area) NODC, Russia

  18. User data access interface – via Integration Server Resource catalogue Parameters Additional search options NODC, Russia

  19. User Data Access interface (cont.) Search status NODC, Russia

  20. User data access interface (cont.) Point data profile data grid data • E2EDM Web site http://data.meteo.ru/e2edm/ • E2EDM Integration Server - IODE Project Office (VLIZ) http://e2edm.vliz.be - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/ NODC, Russia

  21. E2EDM using for SeaDataNet Web-portal User access Registry Discovery Delivery E2EDM Integration Server HTTP communication Data providers community Request message E2EDM Data Provider Connection Encoding Access Responce message NetCDF transport data-file Object transport data-file Metadata@Data SDN Data Centres Network Local Database system Local Data file system NODC, Russia

  22. E2EDM adaptation for SeaDataNet • JRA2 - Technical development of the interoperable system: • E2EDM covers many of JRA2 tasks, but all actions & decisions should be discussed. E2EDM model deepness is enough for many of the SeaDataNet positions. ( + - there is solution in E2EDM) • Discovery services • Metadata update & maintenance + • Potential decision: CDI adoption. • Security services • Data access level + • Transferred data size level + • Delivery services • Downloading services + • Viewing services • Developed user interface, libraries, external tools (based on NetCDF) + NODC, Russia

  23. Thank you! Questions? NODC, Russia

More Related