1 / 19

Overview of the ODP Data Provider

Overview of the ODP Data Provider. Sergey Sukhonosov National Oceanographic Data Centre, Russia. Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011. Scope.

goro
Download Presentation

Overview of the ODP Data Provider

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of the ODP Data Provider Sergey Sukhonosov National Oceanographic Data Centre, Russia Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

  2. Scope Data Provider provides access to data and metadata of the local data systems. When the wrapper is installed in the local data system, the latter becomes a data source for the distributed data system. The Data Provider processes the local data sets and in a semi-automated way generates the discovery metadata. These services are based on the OpenDAP data (point, profile and grid) structures and specific metadata model based on ISO 19115. Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

  3. How to become a data provider The data center which agrees to be the ODP data provider should provide: • the middleware for communications: application server available for HTTP protocol, • installation of the ODP Data Provider software OR use light Data Provider extension, • registration of the data source and its discovery metadata, • support of the local data system Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

  4. How to become a data provider Light Data Provider extension: • Allows integration of data from data centres unable to install the Data Provider software. In this case the owner of the Data Provider must create new user with login and password and provide this information to remote user • Data centres can use remote Data Provider for providing catalogs of data to ODP distributed system

  5. Supported data storage types • Data in relational Database Management System (DBMS): Oracle, MS SQL Server, MySQL, PostgreSQL, etc.; • Structured data files with non-hierarchical data formats (CSV, TSV, …) • Object data files: documents, images, data which has format not supported by ODP technology • Links (web site pages, web-applications, URLs, web-services) Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011

  6. How to provide the data • Data in database inside local network with the Data Provider software • Structured data files: upload to the Data Provider server or specify URL to data files location (FTP, HTTP) Data Provider FTP, HTTP Local network of Data Centre files DBMS Internet Expert training on the Ocean Data Portal technology, Buenos Aires, Argentina, 05-08 July 2011 6

  7. Software architecture To install and operate the ODP Data Provider, it is recommended to use a computer with the following minimum characteristics: CPU 1GHz or more, 2 GB RAM, 300 Mb hard disk space. It is recommended to use a dedicated computer for Data Provider installation. JBoss Application server Data Provider web application Apache web-server PHP Database access serviceDiGIR Dedicated Server running under Windows or Linux OS DBMS Structured and objective data files 7 7

  8. Network requirements • HTTP and SOAP protocols must be available • JBoss AS port must be opened in firewall settings • IP-address verification: Data Provider receives requests only from the Integration Server

  9. Software requirements Middleware software • J2SDK 6 • Application server JBoss 4.2.3.GA (or later) • Web server Apache 2.2.x (or later) • PHP 5.x (or later)

  10. Data exchange protocol • Request and response messages: XML document via HTTP or SOAP protocol • Transport data file : NetCDF file for data from DBMS and structured files, data file “as is” for object data files.

  11. Functional requirements The local data administrator should provide: • design of resources; • data source registration; • discovery metadata registration; • provision of data

  12. How to prepare metadata (design of resource) This work can be handled before or at once after the Data Provider installation and includes the following actions: • local data structure analysis • assessment of resource contents • assesment of data granularity • discovery metadata content (title, area coverage, temporal extent, …)

  13. Local data storage types: SQL-oriented Database Management System: Plain table structure – 1 parameter in 1 column. Local data structure analysis Joining more than couple of tables is not a clever solution – use views if it’s possible

  14. Plain structured data files (CSV, TSV, etc.). List of formats is extendable by adding new data parser; Structured data file can include comment lines, different data value separator; Objective files – no restrictions. Object files must be accessible via HTTP or uploaded to the Data Provider server. Local data structure analysis //M4321;M4322;M4401;M4400;M4050;M4311;M4312;M4303;P0696_00;P0735_00;P0964_00;///P0507_00;P0001_00; //Forecast from 00 hr 20.05.2009. Producing center - NCEP/NOAA. 0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.0000;0;0.6;-9.1;0.01;-37.3;-36.6; 0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.3125;0;0.7;-9.1;0.01;-37.3;-36.6; 0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.6250;0;0.8;-9.1;0.01;-37.2;-36.6; 0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;0.9375;0;0.8;-9.1;0.01;-37.2;-36.6; 0.3125;999.;2009-05-20 00:00;2009-05-20 12:00;0;-89.761;1.2500;0;0.9;-9.1;0.01;-37.2;-36.6;

  15. Resource granularity The Data Provider has the possibility to support various types of the data granularity, i.e. it is possible to create discovery metadata, search and deliver full datasets or specified data pieces (logical data units) of datasets - single cruise or data profile, single buoy or single coastal station data - and etc. The resource can be presented as a single unit (called a single resource) or as a set of resource instances (called a serial resource) reflecting the local data granularity of the local data system. The data granularity level is adjusted in the process of the local dataset registration.

  16. Resource life-cycle • define the schedule for updating of discovery metadata, • check the local dataavailability using the report submitted by Integration Server, • take the needed actions to provide the data source actuality(connection, data files storage availability),

  17. How to generate metadata Use web-interface of the Data Provider to generate, update and maintenance discovery metadata

  18. Case studies • Connecting data to the Data Provider (object files, structured data files, SQL-databases, HTTP location) • Creating new discovery metadata • Creating new discovery metadata using existing one as a template • Managing users of the “Light” Data Provider • Connecting uploaded data via “Light” Data Provider

  19. Thank you for attention! www.oceandataportal.org http://odp.oceandataportal.net JOIN OCEAN DATA PORTAL! HAVE YOU SHARED YOUR DATA TO OCEAN DATA PORTAL?

More Related