1 / 26

TECO-WIS

TECO-WIS. JCOMM E2EDM Pilot Project. Nickolay Mikhailov, JCOMM/IODE ETDMP chair, Russia Sergey Belov, RNODC/RIHMI-WDC, Russia. 6 - 8 November, Seoul. Contents. JCOMM/IODE ETDMP Pilot Project scope Concept, Functionality, Architecture Implementation result E2EDM within WIS E2EDM future.

gusty
Download Presentation

TECO-WIS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TECO-WIS JCOMM E2EDM Pilot Project Nickolay Mikhailov, JCOMM/IODE ETDMP chair, Russia Sergey Belov, RNODC/RIHMI-WDC, Russia 6 - 8 November, Seoul

  2. Contents • JCOMM/IODE ETDMP • Pilot Project scope • Concept, Functionality, Architecture • Implementation result • E2EDM within WIS • E2EDM future

  3. JCOMM/IODE ETDMP Expert Team for Data Management Practices • ETDMP was organized by IOC/WMO JCOMM (JCOMM-1, 2001) • IODE GETADE have been merged with JCOMM ETDMP (2003) Terms of Reference • recommend and implement of end-to-end data management system for JCOMM/IODE • review and assess the effectiveness of data management practices, recommend best practices

  4. ETDMP Pilot Projects 1 - Metadata management • to develop and provide practical testing of a comprehensive metadata model 2 - Data Assembly, Quality Control and Quality Assurance • to examine a scheme of assigning a unique tag to original data, and the variations in QC procedures and quality flags for a various disciplines 3 - E2EDM Prototype • to build and demonstrate a prototype system for real-time data fusion from distributed sources into sample products of interest to JCOMM/IODE users

  5. Pilot Project scope E2EDM Prototype should provides functionality for: • local data systems “wrapping” by E2EDM services • data extraction from distributed sources on user requests/scheduling • on-line aggregation and visualization • creation of new products based on the obtained data E2EDM Prototype should integrates the data: • at operational and delay-mode time scale • across multidisciplinary ocean and marine meteo datasets/flows • from geographically distributed data sources existing in different formats (codes, dictionaries and etc.)

  6. Pilot Project participants • RIHMI-WDC/NODC, Russia • Flanders Marine Institute (VLIZ), Belgium • IFREMER, France • Met Office, UK

  7. E2EDM Concept • Data model • Metadata/data objects specifications • Namespace • Data exchange protocol • Software components

  8. E2EDM data model • Providing identification and formalization of features for main model E2E objects: • Data source • Resource • User (external software application) • Model is based on the following structured semantic objects: • Element • Class - set of elements • Record - set of classes • These semantic objects are used for: • Data granularity and description for local and transferred data • Metadata description • Codes and dictionaries unification • Data sources and users interaction

  9. E2EDM metadata specifications E2ESearchMD – discovery metadata E2ELinkMD – data physical location registry E2EElementsMD – managed data/metadata elements E2ECodesMD – code lists, controlled dictionaries/vocabularies E2EUserMD – end-user external application registry E2EInterfaceMD – end-user interaction and data delivery specifications (interaction type, scheduling, data formats and delivering mechanisms – push/pull)

  10. E2EDM data specifications • E2EDM datarecords: • E2EDataPoint – data with fixed spatial (geographical, depth/height) coordinates, point data type • E2EDataProfile – data with fixed geographical and temporal coordinates, and depth (height) coordinates, profile data type • E2EDataGrid – data distributed in the scope by definite geometrical model

  11. E2EDM data objects specifications (II) E2EDM classes for data granularity and NetCDF data constructions relationship: <E2EObjectHierarchy> <objectHierarchyLevel>1</objectHierarchyLevel> <objectHierarchyList>objectSystemId, objectHierarchyId, countryId, organizationId, dateTime, Latitude, Longitude, platfType, platformId, platformName</objectHierarchyList> <objectHierarchyId>platformId</objectHierarchyId> <instanceId>platformId</instanceId> </E2EObjectHierarchy> <E2EObjectHierarchy> <objectHierarchyLevel>2</objectHierarchyLevel> <objectHierarchyList>DEPH, TEMP, PSAL</objectHierarchyList> <objectHierarchyId>dateTime, Latitude, Longitude</objectHierarchyId> <instanceId>dateTime</instanceId> </E2EObjectHierarchy>

  12. E2EDM namespace • Within E2EDM consists of two sub-namespaces: global and conceptual (thematic). • Global namespace – structures for the E2E data model objects • Conceptual (thematic domain) namespace – list of thematic elements which are available for discovery and exchange (E2EElementsMD) • Both designed in XML Schema. • Global XML Schema (ISO 19115 + WMO Core Metadata + extension) • Concept XML Schema • XML Schemas are also used for validation and development issues (XML Schema -> UML -> software design)

  13. E2EDM data exchange protocol <request> * REQUEST MESSAGE EXAMPLE * <header> <version>0.95</version> <sendTime>2005-08-03T14:47:52+03:00</sendTime> <source>data.meteo.ru</source> <destination resource="RU_RIHMI_01">http://data.meteo.ru:80/digir/DiGIR.php</destination> <type>search</type> </header> <search> <filter> <and> <lessThanOrEquals> <e2edm:Latitude>78.54</e2edm:Latitude> </lessThanOrEquals> </and> </filter> <records limit="100" start="0"> <structure> <xsd:element name="record" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:complexType> <xsd:sequence> <xsd:element ref="e2edm:dateTime"/> <xsd:element ref="e2edm:DEPH"/> <xsd:element ref="e2edm:TEMP"/> </xsd:sequence> </xsd:complexType> </xsd:element> </structure> </records> <count>false</count> </search> </request> <responseWrapper> * RESPONSE MESSAGE EXAMPLE * <response xmlns="http://digir.net/schema/protocol/2003/1.0"> <header> <sendTime>2005-09-29T15:35:17+04:00</sendTime> <source resource="RU_RIHMI_03"/> <type>search</type> </header> <content> <record> <E2ETransportSpecification> <transportFileURL>http://data.meteo.ru:8090/dpms/ncResults/1127993716743-result.nc </transportFileURL> </E2ETransportSpecification> <count>7</count> </record> </content> </response> </responseWrapper> Data exchange protocol is responsible for data requesting and data retrieving. Request message (XML) Response message (XML) Transport data file (NetCDF,v.2.2)

  14. E2EDM software components • E2EDM Data Provider (DP) – set of services, providing interface between local data systems and distributed system. Must be plugged on the local system. Services are using common library (E2EDM API). DP supports data bases, structured and object data files • E2EDM Integration Server – set of services, providing system monitoring, security issues, resource catalogue maintenance and communication with DP (discovery, requesting and retrieving data)

  15. Integration Server. Functions • Metadata update & maintenance • Resource catalogue management • Discovery services for data, metadata and common codes • Request/response communication service • Data sources monitoring • Downloading services (for obtaining data files, metadata, resource and instance descriptions synchronization) • Export tools (e.g. export E2EDM resource description into WMO Core Profile, to ASCII) • HTTP GET/POST (REST) and Web service interfaces • Security service • Schedule mechanisms • Administration tools (CRM, CMS)

  16. Data Provider. Functions • Data source/resources registry and metadata update • Codes and dictionaries mapping • Request handling • Access to the local data source (DBMS, files) • Data extraction based on search criteria • Transport data file generation • HTTP GET/POST (REST) and Web service interfaces • Security service • Schedule mechanisms • Administration tools (online maintenance for resource and instances)

  17. E2EDM architecture Response message netCDF transport data-file Object transport data-file JCOMM/IODE Data Portal User access Security Registry Discovery Delivery Monitoring E2EDM Integration Server HTTP communication Web-services Data providers community Request message E2EDM Data Provider Connection Mapping Data Access Data Data Centers Network Local Database system Local Data file system

  18. Implementation results VLIZ, Belgium MetOffice,UK IFREMER, France NODC, Russia

  19. Implementation results (II) The following data sources are involved in the E2EDM prototype • Historical marine meteorological data • Historical ocean cruise data • Real-time GTS ocean (BATHY and TESAC) data • Real-time GTS ocean (TESAC/ARGO data) • Real-time GTS (SHIP) data • Monthly climatic fields of ocean parameters (imageries) • Ocean SST satellite data Geographic area – North Atlantic

  20. Administrative tools

  21. E2EDM web-site JCOMM/IODE Data Portal link http://e2edm.meteo.ru/ • Project overview • Documentation • Software components • (coming soon)

  22. Current JCOMM/IODE Data Portal Resource catalogue Parameters Requests status Additional search options

  23. Current JCOMM/IODE Data Portal interface (II) Point data Profile data Grid data Object files

  24. E2EDM within WIS E2EDM Integration Server GISC catalogue / registry Harvester « B » OAI DR VGISC protocol communication E2EDM Data Providers DCPC Obninsk « A » WMO and E2EDM metadata records

  25. E2E future implementation ETDMP IODE/JCOMM Ocean Data Portal Scheme for regional/task-oriented approach • IOC IODE – XIX • IODE/JCOMM Ocean Data Portal • ODIN Black Sea Community end-users Global Ocean Data Portal (linked with E2E Integration Server) Regional Ocean Data Portal Regional Ocean Data Portal Themathic Ocean Data Portal Users Users Users Data Provider Data Provider Data Provider Data Provider Data Provider Data Provider Data Provider Data Provider Data Provider Data Provider Data sourcesfederation Data sources federation Data sources federation

  26. Thank you! Questions?

More Related