1 / 23

Common Data Index CDI V1 How to proceed By Dick M.A. Schaap – technical coordinator

Common Data Index CDI V1 How to proceed By Dick M.A. Schaap – technical coordinator Madrid, March 09. S eaDataNet Delivery Services – CDI V1.

hope-combs
Download Presentation

Common Data Index CDI V1 How to proceed By Dick M.A. Schaap – technical coordinator

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Common Data Index CDI V1 How to proceed By Dick M.A. Schaap – technical coordinator Madrid, March 09

  2. SeaDataNet Delivery Services – CDI V1 • Central and homogeneous metadatabase plus transparent access to the distributed data sets via a unique user interface at the SeaDataNet portal and downloading services

  3. SeaDataNet CDI V1 system components • CDI V1 Metadatabase and User Interface: • Searching and browsing of metadata of data sets, managed by SeaDataNet Data Centres • Central User Register - Web service: • Details of users, their organizations and addresses, license period, Id-Passwords, SeaDataNet Roles • Shopping Basket – User Interface: • Preparing a user request of multiple data sets, handling validation of users and roles via the Central User Register, and routing requests to the Request Status Manager • Request Status Manager – Web service + User Interface: • Processing and administration of all requests and data deliveries (downloads), for users in communication with data centres. For data centres to oversee all transactions.

  4. SeaDataNet CDI V1 system components • Data Centre Download Manager – Java Tool: • Handles communication with Request Status Manager and takes care that requested files are made ready for downloading by User (if OK) in the agreed formats via a local website address at the Data Centre

  5. SeaDataNet CDI V1 – implementation step by step • Prepare CDI V1 XML test records using the new MIKADO software and the latest CDI XML schema • Consider carefully the Data Access Restriction, that you want to apply to the data • Send the test records to the XML Validation Services for validation • If ok, then forward the test records to MARIS for inclusion in the CDI V1 test database

  6. SeaDataNet CDI V1 – implementation step by step • Install the Download Manager software; 2 possible situations • You have a file management system • You have a database management system • In case of file management system: Convert data files to SeaDataNet standard formats • Fill the ‘coupling table’ to connect the CDI V1 records (via LOCAL_CDI_ID) to the associated files (by name OR by query) • Test together with MARIS the set-up and good functioning of the connection via the Test interface

  7. SeaDataNet CDI V1 – implementation step by step • If ok, then make full production of CDI V1 records, complete coupling table and make sure that the files are ready / the queries are ok. • Possible interim step: full CDI V1 production in the Import system for checking the quality of the CDI V1 records • Final step: MARIS moves the production CDI V1 records to the operational CDI V1 system and the data centre is active. • Maintenance and updating; only modifications => LOCAL_CDI_ID is important key to recognise updated or new CDI records

  8. SeaDataNet Prepare CDI V1 XML test records • CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record). This is in contrast to EDMED, which is an inventory of data sets, each of which comprises a large number of measurements. • CDI XML Schema is based upon the ISO 19115 standard • At the SeaDataNet website you can find: • CDI V1 XML Schema • CDI V1 XML example • CDI V1 logical description in XLS file • CDI V1 XML files can be prepared with the MIKADO software

  9. CSR EDMED EDMERP CDI [EDIOS] Manual XML files for SeaDataNet catalogues MIKADO Java code Automatic DATABASE JDBC Java DataBase Connectivity EXCEL File Native Drivers MYSQL ORACLE POSTGRES SQLServer Bridge Drivers using Microsoft ODBC (ACCESS, EXCEL, SQL SERVER) Other Drivers Downloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory) SeaDataNet MIKADO Java tool • Available under multiple environments : • Microsoft : Windows 2000, XP, VISTA • APPLE • Unix - Solaris • Linux

  10. SeaDataNet – Consider the Data Access Restriction • User must register once in order to get a personal login - password • Web form to provide necessary information • User agrees with “SeaDataNet User Licence” • After processing, login/password sent by email (email check) • Licence is part of the SeaDataNet Data Policy • NODCs intermediate in the registration of their national users to the SeaDataNet central user register => users will get a user role • Role together with the data access restriction determines the access: • User will have immediate access to the data • The User request will be considered by the Data Centre and possibly the user will be contacted • User will be denied access

  11. User Roles in AAA Register, of relevance for data access Multiple roles may apply See Vocabulary for definitions

  12. Data access restrictions in CDI V1 See Vocabulary L081 for definitions Recommended to use: SeaDataNet Licence!

  13. Business Matrix TBD NO TBD OK TBD TBD OK NO NO NO TBD OK Recommended to use: SeaDataNet Licence!

  14. SeaDataNet – Test CDI V1 records imported • Send the test records to the XML Validation Services for validation • http://www.seadatanet.org/validator • If ok, then forward the test records to MARIS for inclusion in the CDI V1 test database http://seadatanet.maris2.nl/v_cdi_v1_test/search.asp http://seadatanet.maris2.nl/v_cdi_v1_test/browse_step.asp

  15. SeaDataNet CDI V1 – Data Access example Check Status In RSM Search Request Confirmed Include in Basket Results Ready at DC x Download Shopping list Data SDN format Submit + Authentication

  16. SeaDataNet Download Manager V1.1 Installation • Install and configure the Download Manager V1.1 software, following the installation instructions and in discussion with MARIS, who has to register the new data centre in the RSM configuration. • At the SeaDataNet Extranet (BSCW) in the Technical Task Team folders you can find all the relevant documentation and tools for the upgrade: • CDI V0 – V1 Steps guidance document (see Delivery Services folder) • Specifications of the Download Manager (see Delivery Services folder) • Technical documentation of the Download Manager (see Delivery Services folder) • Installation instruction of the Download Manager (see Delivery Services folder) • Software package of the Download Manager V1.1d (see Delivery Services folder) • Transition document if you go from V1.1c to V1.1d (see Delivery Services folder)

  17. SeaDataNet Download Manager V1.1 Installation • Install the Download Manager software; 2 possible situations • You have a file management system • You have a database management system • In case of file management system: You have to convert data files to SeaDataNet standard formats • In case of database management system: Conversion is done ‘on the fly’ by the Download Manager software • Fill the ‘coupling table’ to connect the CDI V1 records (via LOCAL_CDI_ID) to the associated files (by name OR by query) • Test together with MARIS the set-up and good functioning of the connection via the Test interface

  18. SeaDataNet Download Manager V1.1 Functions • Communication with RSM Web service about process progress • Processing order lists => retrieving requested data files • Retrieving data sets from: • file management systems • database systems (e.g. Oracle, MySQL, MS-SQL, Sybase, ….) • If data sets are retrieved from a database => export of data files in SDN ODV4 format • Note: Data sets from a file management system must be pre-processed in SDN transport formats (ODV4 and optional MedATLAS for ASCII type data, or NetCDF (CF) for grid type data) • Combining data set files in daily zip files in a user download directory • Generating dynamic user download webpage, following SDN template, to give user overview and option of downloadable zip files

  19. SeaDataNet – Converting to SDN Data Formats • V1: data sets are accessable by downloading services. Delivery of data sets to users requires common data transport formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and analysis & presentation tools (ODV, DIVA) • The following formats have been defined: • SeaDataNet ODV4 ASCII for profiles, time series and trajectories • SeaDataNet MedAtlas as optional extra format. • NetCDF with CF compliance for gridded data sets • ODV4 and MedAtlas have been outfitted with a SeaDataNet semantic header • NEMO tool ready for conversion of ASCII files to SeaDataNet ODV and MedATLAS • Med2MedSDN V1.0 ready for conversion of MEDATLAS files without the SeaDataNet extensions to MEDATLAS files with SeaDataNet extensions (SeaDataNet MEDATLAS)

  20. SeaDataNet CDI V1 – next steps • If the DM – RSM connection works fine in the Test system, then you are ready to move to full production and operational status • Make full production of CDI V1 records • Complete the coupling table • Make sure: • associated files are ready OR • all SQL queries are ok. • Possible interim step: full CDI V1 production in the Import system for checking the quality of the CDI V1 records • http://seadatanet.maris2.nl/v_cdi_v1_import/search.asp • http://seadatanet.maris2.nl/v_cdi_v1_import/browse_step.asp

  21. SeaDataNet CDI V1 – final steps • Final step: MARIS moves the production CDI V1 records to the operational CDI V1 system and the data centre is active. • http://seadatanet.maris2.nl/v_cdi_v1/search.asp • http://seadatanet.maris2.nl/v_cdi_v1/browse.step • Maintenance and updating; only modifications => LOCAL_CDI_ID is important key to recognise updated or new CDI records

  22. SeaDataNet CDI V1 – Quick Search interface Using an intuitive approach which is easy for users

  23. SeaDataNet CDI V1 interim as temporary alternative • All Data Centres must populate the CDI V1 database: • Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software • V1 interim = no installation of Download Manager, but data centres will be informed by the Request Status Manager by e-mail of all requests • Data centres have to process all requests manually using the Request Status Manager • Delivering data sets in SeaDataNet Formats (conversion via new NEMO tool) – pre-processing required. • Effect is, that users will experience a common approach to all data centres. The difference is behind the scenes. • Thereafter centres should move to full V1 status by installing DM

More Related