CCSDS - MOIMS Area Data Archive Ingest WG CNES Report Colorado Springs meeting – January 2007

1. CCSDS - MOIMS AreaData Archive Ingest WGCNES Report Colorado Springs meeting – January 2007 Claude Huc

2. Contents • CNES standards, evolution of data engineering domain (RNC) • A new data standard format for the CDPP • Current work on the ‘Producer Archive Interface Specification’

3. CNES standards (RNC)current content of the data engineering domain • At the highest level • An introductory document justifying its existence • The OAIS reference model • On a technical level • A document defining Criteria for evaluating data formats • The DEDSL abstract standard • The DEDSL XML/DTD syntax • The EAST specification • The PAIMAS • Applicable rules and recommendations for projects producing data • New document planned in 2007 : • Long term archiving of data: Applicable rules and recommendations for archive services

4. A new data standard format for the CDPP - Plasma Physics data Center - Main existing data standard formats in space physics community : • CDF – Common data Format • A very good format for data analysis • Number of general graphic, statistic and analysis tools available • Efficient for access and file size • however • Not really powerful for the metadata definition and management • Representation information apparently not fully public and quite complex • Used more for summary data than for high resolution data • Seems not really appropriate for long term preservation

5. A new data standard format for the CDPP Main existing data standard formats in space physics community • Cluster Exchange Format • Almost the high resolution data in the Cluster Data Archive at ESTEC are CEF conformant • A significant number of the space physics area are covered • However • Specific syntax for the metadata • Ascii files with impact • On file sizes • On access performances • Other used formats : NetCDF, FITs… • In practice : no standard format in the plasma physics community • Consequences for the CDPP : about 200 high resolution data sets and about 150 different data format

6. Main objectives for a new CDPP format • A Format for the long term preservation • A format allowing multiple and generic added value services • To make easier the data access • To make easier the data analysis • Allowing simple field extraction • Make easier the creation of composed parameters ( alfven velocity,…) • …

7. The metadata • A data set conformant with the CDPP format specification includes • A set of files • One XML Metadata • These metadata are base on the CDPP and CAA (Cluster Active Archive) dictionaries • The XML metadata file describes • the parameter level metadata (semantics and syntactic features) • General parameters of the files in this data set : type of coding (binary/ascii), type of date/time…) • The metadata can be extended without impact on the data files • Metadata block structure : syntax, parent-child relation, scientific interpretation…

8. Main features of the data files • A very simple structure under the form of a temporal series • Standard (10-6 sec) or precise (until 10-12sec) dates and time • Binary or ASCII files (defined in the XML metadata file) • Management of scalars, vectors, multidimensional arrays . • Management of variable size arrays

9. Associated tools • Read and write Java and C libraries • ASCIIBinary transformation • Toof for creation new data sets conformpant with CDPP format from existing data • Tools currently under development • translation CDFCDPP. • translation CEFCDPP. • Translation NetCDF  CDPP • Tool to make easier the metadata file creation Les outils en cours de développement • Convertisseur CDFCDPP. • Convertisseur CEFCDPP. • Outil d’aide à la constitution du fichier de métadonnées.

10. Ongoing work for Producer Archive Interface Specification • The document is currently being thoroughly revised • Progressive convergence between NASA and CNES for the concepts and the terminology • See Daniele’s presentation

11. The end…