Data preservation Using CCSDS and ISO Standards

1. Data preservationUsing CCSDS and ISO Standards Denis MINGUILLON denis.minguillon@cnes.fr ERPANET - Marburg 2003

2. The problem to solve • Data is often expensive to produce • Bits must not be lost (preserved medium) • Bits must remain accessible (known syntax) • Bits must remain understandable (known semantics) • This presentation focuses on point 2 and 3 ERPANET - Marburg 2003

3. The syntactic aspect • How to master the bit organisation • 2 possible solutions • Define exchange formats • Better done by user communities (ex : FITS for astronomy) • Not generic enough • Define a standard way for describing any data • Was the CNES proposal to CCSDS • Was entrusted to CNES (definition and tools) by CCSDS ERPANET - Marburg 2003

4. CCSDS recommendations History • CNES proposed a description language • EAST for data syntax • CCSDS recommendation since May 1997 CCSDS 644.0-B-2 Issue 2 November 2000 • ISO standard since 2000 ISO 15889-2000 ERPANET - Marburg 2003

5. Data syntax description EAST principles • EAST is a standard language • Enhanced Ada SubseT • inherits from a huge standardisation work • enhanced with a few features missing for data description • EASThas not to be mastered to be used • thanks to the tools ERPANET - Marburg 2003

6. Data syntax description EAST principles(2) • Assumption : any kind* of digital data (available as an ordered set of bits) can be described as a tree. • The minimal tree is a rake *note : optimal for binary data, interesting for raw ASCII format, of no use for XML data • To describe such a tree, EASTproposes : • composite types for the branches • elementary types for the leaves ERPANET - Marburg 2003

7. Data syntax description EAST data types • Composite types are : • records • arrays • lists • Elementary types are : • integers and reals • enumerated • characters ERPANET - Marburg 2003

8. Data syntax description Optionality • Some branches or leaves of the data tree may be optional. • EAST offers a way to indicate which data entities are optional and under which condition they exist • Example : • Housekeeping data exists if telemetry mode is set to « housekeeping » ERPANET - Marburg 2003

9. Data syntax description Variable size • Some repetitions of data entities may have a variable number of elements • EASToffers a means to indicate how many of those elements are present in the current data ERPANET - Marburg 2003

10. Data syntax description EAST use conclusion • All the features described in the previous slides have been sufficient to describe almost every data encountered since 1997 • Exceptions : • variable size not carried by the data • variable size provided after the variable size field ERPANET - Marburg 2003

11. CCSDS recommendations History (2) • CNES has also been in charge of writing the recommendation for data semantics description : • DEDSL (Data Entity Dictionary Specification Language) • XML/DTD syntax • CCSDS recommendation since January 2002 CCSDS 647.1-B-1 Issue 1 January 2002 • ISO standard since 2003 ISO 22643 : 2003 ERPANET - Marburg 2003

12. Data semantics description DEDSL principles • The data semantics description can be kept as a set of attribute values. • The recommendation defines standard attributes : name, definition, units…. • The recommendation gives a standard way to define user attributes to be added to the standard ones if needed ERPANET - Marburg 2003

13. DEDSL principles (2) • A data description consists in the tree organisation (as seen on the sysntactic aspect) completed by all the fields (nodes or leaves) caracteristics given as attributes values. • Example of the descripon of a leaf of such a tree description. ERPANET - Marburg 2003

14. A tool to describe data OASIS • OASISis the tool to describe data (syntax and semantics) in a friendly way (without any requested knowledge of the EAST or DEDSL syntax) • A dedicated Man-Machine interface offers the requested features (those listed previously) to describe data ERPANET - Marburg 2003

15. A tool to describe data OASIS (2) • Example : • a data tree as built and shown by OASIS ERPANET - Marburg 2003

16. A tool to describe data OASIS (3) • Once the tree and each of its nodes and leaves described, the descriptions can be produced just by a click (EASTand DEDSL-XML files) • Once saved, the EAST description can be used by the EAST tools to read and write the described data • Once saved, the XML dictionary can be published as an html or word document ERPANET - Marburg 2003

17. Tools for EAST described data EAST tools • Application Programming Interface (API) • GENERATOR : for data writing • INTERPRETER : for data reading • Both available for C, C++, FORTRAN and Ada • Standalone tools (based on the API) • data_checker : data vs description compliance • ascii_dump : display any data in ascii values • DUW : interactive data generator • other utilities… ERPANET - Marburg 2003

18. Tools for EAST described data EAST tools (2) • EAST tools are available on : • PC (DOS, NT, Linux), • HP-UX, • SUN-Solaris, • Dec true64 (OSF1) , • VAX-VMS • other on request…. ERPANET - Marburg 2003

19. Documenting the interfaces Interface Control Documents • The DEDS-XML OASIS output can be transformed in an html or Word document • A tool (named SCRIBE) is being developed for that purpose . ERPANET - Marburg 2003

20. OASIS tool EAST Apis DCI generation File descriptions EAST ‘RAW’ Data XML applications Html or Word document XML Data Synthesis ReadWriteGenerateCheck Data dictionary DEDSL/XML ReadWriteGenerateCheck File descriptions XML schema ERPANET - Marburg 2003

21. The project conclusions Main advantages • Only one consistent reference for syntactic description used for input/output and for semantic description (documentation for understanding) • Reusable and reliable generic input/output software available • User application easier to maintain due to less impact from changes of data formats • Help in describing data and producing documentation ERPANET - Marburg 2003

22. Use in other projects EAST/DEDSL-XML in action • Many projects at CNES • SPOT, SSALTO, SIPAD, HELIOS, PHARAO… • ESA projects • AMS, DEBAT • NASA • Science data rehabilitation ERPANET - Marburg 2003

23. Projects feedback EAST/DEDSL-XML feed back • PROs : • Easier sub-system integration • description accuracy (even after long time!!!) • early data simulation • assisted data checking • Easier ICD evolutions • full top-down process • Helpful in heterogeneous platforms context • automatic conversions • Helpful in multi-missions context ERPANET - Marburg 2003

24. Projects feedback EAST/DEDSL-XML feed back (2) • CONs : • Performances to be again improved • reading in time constrained context • writing in any case • User type sharing to be added • to facilitate evolutions • Documentation production to be improved • project customisation • add as much filters as possible to allow adaptation of the document to its target ERPANET - Marburg 2003

25. On going activity Current actions • Documentation tool in progress • generic tool to transform XML-DED in Word or html documents • 2nd prototype delivered • Main lacks taken into account by the DEBAT project (ESA/ESRIN) • Support to NASA • tests on data migration - good first results • Support to XML schema production ERPANET - Marburg 2003

26. Information CNES policy for the EAST tools • OASIS and EAST tools are freely available • They can be downloaded from a CNES server • Tools are maintained by a CNES team and a sub-contractor team (CS-SI) • Policy : if CNES stops founding the maintenance the source code would be made available ERPANET - Marburg 2003

27. Information Links • For any support request : • east@cnes.fr (e-mail address) • For information and documentation • east.cnes.fr (web server) • To download the software : • logiciels.cnes.fr (web server) • To get the standards • http://www.ccsds.org (web server) • then under publications, recommendations and reports, information access and interchange systems. ERPANET - Marburg 2003