1 / 11

CCSDS - MOIMS Area Data Archive Ingest WG CNES Report Colorado Springs meeting – January 2007

CCSDS - MOIMS Area Data Archive Ingest WG CNES Report Colorado Springs meeting – January 2007. Claude Huc. Contents. CNES standards, evolution of data engineering domain (RNC) A new data standard format for the CDPP Current work on the ‘Producer Archive Interface Specification’.

elma
Download Presentation

CCSDS - MOIMS Area Data Archive Ingest WG CNES Report Colorado Springs meeting – January 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CCSDS - MOIMS AreaData Archive Ingest WGCNES Report Colorado Springs meeting – January 2007 Claude Huc

  2. Contents • CNES standards, evolution of data engineering domain (RNC) • A new data standard format for the CDPP • Current work on the ‘Producer Archive Interface Specification’

  3. CNES standards (RNC)current content of the data engineering domain • At the highest level • An introductory document justifying its existence • The OAIS reference model • On a technical level • A document defining Criteria for evaluating data formats • The DEDSL abstract standard • The DEDSL XML/DTD syntax • The EAST specification • The PAIMAS • Applicable rules and recommendations for projects producing data • New document planned in 2007 : • Long term archiving of data: Applicable rules and recommendations for archive services

  4. A new data standard format for the CDPP - Plasma Physics data Center - Main existing data standard formats in space physics community : • CDF – Common data Format • A very good format for data analysis • Number of general graphic, statistic and analysis tools available • Efficient for access and file size • however • Not really powerful for the metadata definition and management • Representation information apparently not fully public and quite complex • Used more for summary data than for high resolution data • Seems not really appropriate for long term preservation

  5. A new data standard format for the CDPP Main existing data standard formats in space physics community • Cluster Exchange Format • Almost the high resolution data in the Cluster Data Archive at ESTEC are CEF conformant • A significant number of the space physics area are covered • However • Specific syntax for the metadata • Ascii files with impact • On file sizes • On access performances • Other used formats : NetCDF, FITs… • In practice : no standard format in the plasma physics community • Consequences for the CDPP : about 200 high resolution data sets and about 150 different data format

  6. Main objectives for a new CDPP format • A Format for the long term preservation • A format allowing multiple and generic added value services • To make easier the data access • To make easier the data analysis • Allowing simple field extraction • Make easier the creation of composed parameters ( alfven velocity,…) • …

  7. The metadata • A data set conformant with the CDPP format specification includes • A set of files • One XML Metadata • These metadata are base on the CDPP and CAA (Cluster Active Archive) dictionaries • The XML metadata file describes • the parameter level metadata (semantics and syntactic features) • General parameters of the files in this data set : type of coding (binary/ascii), type of date/time…) • The metadata can be extended without impact on the data files • Metadata block structure : syntax, parent-child relation, scientific interpretation…

  8. Main features of the data files • A very simple structure under the form of a temporal series • Standard (10-6 sec) or precise (until 10-12sec) dates and time • Binary or ASCII files (defined in the XML metadata file) • Management of scalars, vectors, multidimensional arrays . • Management of variable size arrays

  9. Associated tools • Read and write Java and C libraries • ASCIIBinary transformation • Toof for creation new data sets conformpant with CDPP format from existing data • Tools currently under development • translation CDFCDPP. • translation CEFCDPP. • Translation NetCDF  CDPP • Tool to make easier the metadata file creation Les outils en cours de développement • Convertisseur CDFCDPP. • Convertisseur CEFCDPP. • Outil d’aide à la constitution du fichier de métadonnées.

  10. Ongoing work for Producer Archive Interface Specification • The document is currently being thoroughly revised • Progressive convergence between NASA and CNES for the concepts and the terminology • See Daniele’s presentation

  11. The end…

More Related