1 / 133

SeaDataNet tools NEMO, MIKADO, VALIDATOR

SeaDataNet tools NEMO, MIKADO, VALIDATOR. M. Fichaut, IFREMER, France. SDN tools directory. On your desktop, SDN tools directory : presentationsSDN_tools.pps. Overview. SeaDataNet tools in their context NEMO : Reformatting tool Practical session : reformatting a set of stations

stokesr
Download Presentation

SeaDataNet tools NEMO, MIKADO, VALIDATOR

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SeaDataNet toolsNEMO, MIKADO, VALIDATOR M. Fichaut, IFREMER, France

  2. SDN tools directory • On your desktop, SDN tools directory : \presentations\SDN_tools.pps

  3. Overview • SeaDataNet tools in their context • NEMO : Reformatting tool • Practical session : reformatting a set of stations • NEMO and SDN Download Manager – coupling table, NEMO and MIKADO – CDI summary file • Practical session : generation of a CDI summary file and of a coupling table • MIKADO : XML files generator, Web validation service, Link between MIKADOand SDN Download Manager – coupling table • Practical session : • Automatic creation of XML files for CDI, Web validation of these files, creation of the corresponding coupling file • Manual input for EDIOS program, series and platform • Automatic creation of XML files for EDIOS

  4. CSR Metadata In database Meta data at XML ISO19155 New metadata Metadata in Excel files EDIOS Coupling table Data in database EDMED SEADATANET PORTAL Collection of ASCII files at SDN Format New data Data in ASCII file Format X NEMO EDMERP Download Manager CDI Local copy of data for downloading AAA Data request RSM Downloading European portal Data centres XML validator SeaDataNet Vocabulary MIKADO MIKADO

  5. ODV DIVA Data Centre CSR Local ID+ File address or SQL Select Coupling table Data in database EDIOS Collection of ASCII files at SDN format Data User Data Central ID EDMED SEADATANET PORTAL Zip file EDMERP Download manager Local ID Data request CDI Local ID + File address Local ID + SQL Select AAA Central ID + ID local + EDMO code Central ID + Local ID+ EDMO code Local copy of data for downloading RequestStatus Manager (RSM) Zip file Downloading

  6. Survey about the tools • 29 answers • NEMO : question about a Cruise name field • Questions on the coupling table : how to generate it, using which tool? • Those who know, please help those who don’t know

  7. NEMO – Reformating software

  8. XML validator SeaDataNet Vocabulary CSR Meta data at XML ISO19155 Metadata In database New metadata MIKADO Metadata in Excel files EDIOS MIKADO Coupling table Data in database EDMED SEADATANET PORTAL Collection of ASCII files at SDN Format New data Data in ASCII file Format X NEMO EDMERP Download Manager CDI Local copy of data for downloading AAA Data request Downloading RSM European portal Data centres : SeaDataNet nodes

  9. Objectives • NEMOis a reformatting software used to generate ASCII files at MEDATLAS or ODV formats which are defined as SeaDataNet formats for data exchange between SeaDataNet partners. Can be downloaded from SeaDataNet Web site : http://www.seadatanet.org/standards_software/software/nemo • It was created in IFREMER for internal purposes, and adapted to take into account the SeaDataNet formats and vocabularies.

  10. Technical characteristics • Release : 1.4.4 • Written in Java Language (Version >= 1.6) • Bilingual (French, English) • Available under multiple environments : • Microsoft : Windows 2000, WINDOWS 7, XP, VISTA , APPLE • Unix - Solaris • Linux • Interactive and batch modes available • Use of SeaDataNet common vocabularies web services • to update lists of values of the SeaDataNet common vocabularies • need network connections in order to have up to date lists of values. • But NEMO works offline once the lists are up-to-date

  11. NEMO main features • Reformat ASCII text file of vertical profiles, time-series or trajectories to a SeaDataNet ASCII format (ODV, MEDATLAS). • The input ASCII files can be: • one file per station for vertical profiles or time series • one file for one cruise for vertical profiles, time series or trajectories • Interact with Mikado, to be able to generate ISO-19115 XML descriptions of the data.

  12. NEMO principles • NEMO must be able to read almost any ASCII format to translate it to MEDATLAS, ODV • Users of NEMO describe the entry files format so that NEMO is able to find the information which is necessary in the SeaDataNet formats • One mandatory pre-requirement is that in the set of entry files the information about the stations must • be located at the same position : same line in the file, same position on the line or same column if CSV format • be in the same format • For example : for all the stations the latitude is • on line 3 on the station header, • from character 21 to character 27, or 3rd column in CSV • the format is +DD.ddd

  13. Running NEMO • 4 or 5 steps to convert the entry files to a SeaDataNet format • Describe the type of file(s) • [Describe the cruise, if the files are related to 1 cruise] • NEMO is able to read XML CSR generated with MIKADO • Describe the station information • Describe the measured parameters • Convert the file • One more step to • Save the description of the format (Model)

  14. NEMO - Description of entry files (1) • Where are the files? • Is it a cruise? • Is it one file per cruise or one file per station ? • Is it a collection of station files ? • grouped by cruises or not? • Are the files with separators? • Tabulations? Semicolon? Comma? • Are they vertical profiles, time series or trajectories data?

  15. NEMO - Description of entry files (2) File to convert is displayed in NEMO window The principle is to describe the input file by selecting the information in the file window

  16. Station header : constant number of lines

  17. Station header : number of lines not constant

  18. Cruise description Can be: Manually input Imported from a SeaDataNet XML CSR Imported from a database (IFREMER only)

  19. Station description

  20. Station description Select the date in the file, in the left part of the screen

  21. Station description Input the dateformat

  22. Station description Click on the Set button to tell NEMO the start and end position of the date

  23. Station description Click on the Test button to verify your input

  24. Station description

  25. Data description Choose the parameter list

  26. Data description Choose the parameter list Give the code, unit, position , format and input default value of each parameter

  27. Data description Select « Parameter list » in the context menu of the measured parameters table (right click to open the menu)

  28. Data description Add all the measured parameters and their unit (P061) using the context menu

  29. Data description Select the value of the measured parameter in the file, in the upper left part of the screen

  30. Data description Select « Set start /end » in the context menu of the measured parameters table

  31. Data description Start and End position of depth are added in the table

  32. Data description Describe the format and default value of the parameters (output default value only if output format is MEDATLAS)

  33. Data description Check your input : Select « Update test » in the context menu of the measured parameters table

  34. Data description The values read by NEMO are displayed in the Test column of the table. User is able to check that his inputs are correct

  35. Data description Tell NEMO which parameter is the vertical reference

  36. Output formats • MEDATLAS • Multi-station • ODV • Mono-station • Multi-station File conversion

  37. For sediment description in Sediment Core Compatible with ODV Non numeric parameters

  38. NEMO in batch mode • MIKADO can be run in batch mode, from NEMO install directory, using existing models • Several arguments can be added on the command line • Log file in NEMO install directory

  39. Files which can’t be reformatted by NEMO • Binary format : like EXEL, WORD … • File have to be first reformatted to text file • Files which do not respect NEMO pre-requirements • be located at the same position : same line in the file, same position on the line or same column if CSV format • be in the same format • Files which are not : • Vertical profiles (with depth or pressure as vertical reference) • Time series (with time as reference) • Trajectories (with latitude, longitude and time as reference)

  40. 1201 310219590650370-029-330100 2 19660719140058999000000000000080161 013 30029800000258+ + + + 201214 42010 28 0000 6000000036043261501461904 6000090030943320801460604 6000240039043409401465404 6000470054343452101472604 6000710047843449101470404 6000940052043472301472704 6001180059643491301476404 6001410065743509201479404 6001650060343506601477704 6001880051943491601474504 6002120052343496101475104 6002340046543491801473104 1201 31025246064008000060530000 2 19660806190058999000000000000170161 013 30030900000298+ + + + 201214 40213 0000 6000000147043307601503504 6000100133643417301500604 6000250119043467401496504 Like for example old historical tape formats

  41. NEMO – User manual • User manual is provided : File :sdn_Nemo_UserManual_V1.4.4.pdf • Detailed explanation for NEMO installation and use, lots of snapshots

  42. NEMO next releases • V1.4.5 (soon) • With correction of some know bugs • V1.5.0 (scheduled in autumn) • BODC vocabulary, version 2 • Pre-requirement : Importation tools must manage BODC V2 vocabulary • Include a Cruise name field that can be read in the input file • V1.5.1 (not scheduled yet) • Generation of files at NetCDF format • Pre-requirement • NetCDF format final definition

  43. Practical work on NEMOConverting files M. Fichaut Training Workshop – Ostende – 1-4 March 2010

  44. ODV files Collection of ASCII files XML CDI files SeaDataNet CDI summary_CDI_NEMO.xml Delivered with NEMO CDI summary CSV file MIKADO From ASCII data to XML files Coupling table Export Coupling table

  45. NEMO installation • Open the install_nemo_1.4.4_windows.zip file (on the desktop) • Unzip it • Double click on launcher_nemo.bat • Follow the installation procedure • Don’t forget to add the NEMO shortcut on your desktop

  46. Collection of ASCII files • Training data set • 6 CTDs raw files under SDN tools/Practical work/data/input files • ASCII files with measured parameters in column • Measured parameters = Depth, Temperature, Salinity, Fluorescence and Dissolved oxygen

  47. First step : conversion to ODV with NEMO • Run NEMO to convert the files to ODV format. • Modify the settings of NEMO • Default directories for data set to SDN tools/Practical work/data • Default directories for data set to SDN tools/Practical work/models • Enter your EDMO ID in the field below using the search function (right click) • Proceed the 4 steps of NEMO

  48. NEMO – File description • Choose the type of file (cruise directory, cruise file, …) • Browse your PC to input the data files directory • Once the file is open in NEMO, have a look to its content • Choose file type : Profile • Choose the output format : ODV, one unique file for all stations or One file per station • Enter the Station header information • there is a specific character string at the end of each Station header : *END* • The Data termination indicator is EOF (End of file) • Validate the step when description is OK

  49. NEMO – Station description • Station number is given on line “**Station” • Data type is ‘CTD stations’ • Time is U.T. • Latitude format is NDD MM.mm • Longitude format is NDDD.MM.mm • Bottom Depth is not available • Input all the mandatory fields : station number, time, date, latitude, longitude • Use the Test function to verify your input • Validate the step when description is OK

More Related