540 likes | 706 Views
Contribution of SAFE to XFDU. CCSDS/MOIMS/IPR Workshop (11-15/04/2005 Athens). Stéphane MBAYE GAEL Consultant Stephane.Mbaye@gael.fr. Rationale. SAFE Format Specifications 1.5 – [SAFE-FMT] ESA/CNES XFDU I/O Library – [XFDU-I/O] SAFE I/O library – [SAFE-I/O]
E N D
Contribution of SAFE to XFDU CCSDS/MOIMS/IPR Workshop(11-15/04/2005 Athens) Stéphane MBAYEGAEL ConsultantStephane.Mbaye@gael.fr
Rationale • SAFE Format Specifications 1.5 – [SAFE-FMT] • ESA/CNES XFDU I/O Library – [XFDU-I/O] • SAFE I/O library – [SAFE-I/O] • XFDU Specifications – Issues and Proposals
SAFE - Origin and goals (reminder) • Standard Archive Format for Europe • Provide ESA with a single format for all EO data • All Level 0 (Raw) and Level 1 • Level 2 (as far as applicable) • Ground Segment Auxiliary files) • Should become an ESA AIP • Candidate for SIP and DIP (TBV) • Shall be compatible with XFDU
SAFE – 111 Data types to be supported NOAA (6)AVHRR/TOVS (6) IRS P3 (1)MOS (1) Nimbus (2)CZCS (2) MOS (1)MESSR (1) SPOT (3)HRV (1), HRVIR (1), HRG (1) • ENVISAT (59)ASAR (23), AATSR (5), DORIS (3), GOMOS (7), MERIS (13), MIPAS (6), MWR (1), RA2 (8), SCIAMACHY (5) • ERS (25)RA (4), AMI (12), ATSR (3), MWR (1), GOME (2), PRARE (3) • Landsat (3)MSS (1), TM (1), ETM+(1), RBV (0) • JERS (3)OPS (1), SAR (2) • Terra/Aqua (1)MODIS (1) • NOAA (6)AVHRR/TOVS (6) • Orbview (5)SeaWifs (5)
SAFE, a restriction of XFDU <!-- SAFE Root element type. The type of the SAFE root derives from XFDU root type. --> <xsd:complexType name="safeType"> <xsd:complexContent> <xsd:restriction base="XFDUType"> <xsd:sequence> <xsd:element form="unqualified" name="packageHeader« type="safe:packageHeaderType" minOccurs="1« /> <xsd:element form="unqualified" name="informationPackageMap" type="safe:informationPackageMapType" maxOccurs="1« /> <xsd:element form="unqualified" name="metadataSection" type="safe:metadataSectionType" minOccurs="1« /> <xsd:element form="unqualified" name="dataObjectSection" type="dataObjectSectionType" minOccurs="0« /> <xsd:element form="unqualified" name="behaviorSection" type="behaviorSectionType" minOccurs="0« /> </xsd:sequence> </xsd:restriction> </xsd:complexContent> </xsd:complexType> <!-- SAFE Root element --> <xsd:element name="safe" type="safe:safeType"/>
XML Schemas / Namespace policy http://www.esa.int/safe/1.0 http://www.esa.int/safe/1.0/core-1.0.xsd http://www.esa.int/safe/1.0/passive-sensor-1.0.xsd http://www.esa.int/safe/1.0/active-sensor-1.0.xsd http://www.esa.int/safe/1.0/impl/xxx Specializations…
SAFE Information Package Map Section One and only one Information Package Map Section
SAFE – Core Metadata Objects • Acquisition Period Object • Platform Objects • Processing Log Objects • Quality Information Objects • Orbit Reference Objects • FrameSet Objects • XML Schemas Objects
SAFE - Specialized Core Objects <!-- Platform description --> <metadataObject ID="platform" classification="DESCRIPTION" category="DMD"> <mdWrap textInfo="Platform Description" vocabluaryMdType="OTHER" otherMdType="SAFE" mimeType="text/xml"> <xmlData> <safe:platform> <!-- Platform identification --> <safe:nssdcId>2003-009A</safe:nssdcId> <safe:familyName>ENVISAT</safe:familyName> <!-- Instrument identification --> <safe:instrument> <safe:familyName>MERIS</safe:familyName> <meris:mode vcid="6" opsMode="4"/> </safe:instrument> </safe:platform> </xmlData> </mdWrap> </metadataObject>
SAFE – Quality Information Use of XPath/XQuery foridentifying the corrupted/ Missing areas of theData Objects <!-- Quality information of the measurements --> <metadataObject ID="measurementQualityInformation" classification="DESCRIPTION" category="DMD"> <mdWrap textInfo="Quality Information" vocabluaryMdType="OTHER" otherMdType="SAFE" mimeType="text/xml"> <xmlData> <safe:qualityInformation> <!-- 24 records are missing from after the 1233rd record --> <safe:missingUnits> <safe:location>mdsr[1234]</safe:location> <safe:count>24</safe:count> </safe:missingUnits> <!-- The records 456 to 468 have a bad application ID --> <safe:corruptedUnits> mdsr[456 to 468]/isp/pkHd/PkId/appId </safe:corruptedUnits> </safe:qualityInformation> </xmlData> </mdWrap> </metadataObject>
SAFE – XML Schema Objects • All Data Objects shall have an associated XML Schemas • Post-processing (sub-setting, reformatting, …) • Annotation (quality information, cloud cover) • Indexing • Documentation / Presentation
SAFE - Preservation of Binary Objects • Annotated XML Schemas • Structured Data File (SDF) markups <xsd:complexType> <xsd:sequence> <xsd:element name="record" type="xs:int" minOccurs="0" maxOccurs="unbounded"> <xsd:annotation> <xsd:appinfo> <sdf:block> <sdf:occurrence query="../header/numRec + 1"/> <sdf:length unit=‘bit’>12</sdf:length> </sdf:block> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> <xsd:complexType/>
SDF - Example of Binary interpreter (Derby) Detailed content of an ENVISAT MERIStelemetry Source Packet(CCSDS standard)described by SDF
Active Sensors Metadata Objects • Pulse Repetition Frequency • Polarization • Pulse Length
SAFE Behavior Objects • No SAFE Behavior Object has been defined • Some may be added for configuring Stitching? • Should be investigated with CNES
Getting XFDU I/O library –Web site http://www.gael.fr/xfdu
XFDU I/O Basic principles • Based on interactions with the MANIFEST file • Each interaction is not automatically comitted (possible rollback): the result shall be explicitely saved • 100% Java • C++ Wrapper based on Java Native Interface (JNI) • No support of Behavior section (to be investigated with CNES)
XFDU I/O Implemented Usecases • Creation of a new XFDU packet • Opening of an existing XFDU packet • Removal of a XFDU packet • Moving a XFDU packet (with or without the attached objects) • Management of the Content Units • Management of Metadata Objects (only XML wrap) • Management of Data Objects (only Flocat references) • Browse Data Object Content (if associated XML Schema) • Validation of XFDU packet • Dumping XFDU Manifest in XML
XFDU I/O – Configuration Status • Configuration identifiers: • safe-1-0-beta-2: first functional release • safe-1-0-beta-1: API only • Package identifier: esa.xfdu (TBV CNES)
Getting SAFE I/O library – SAFE Web site http://www.gael.fr/safe
SAFE I/O Basic principles (Cont’) • Delegates standard XFDU access to ESA XFDU I/O • Based on interactions with the MANIFEST file • Each interaction is not automatically comitted (possible rollback): the result shall be explicitely saved
SAFE I/O Implemented Usecases • Creation of a new SAFE packet • Opening of an existing SAFE packet • Removal of a SAFE packet • Moving a SAFE packet (with or without the attached objects) • Management of the Content Units • Management of the Core Metadata Objects • Adding specialized Metadata • Management of Data Objects • Extraction or pointing to a subpart of a Data Object • Identifying missing units and corrupted areas • Browse Data Object Content • Validation of SAFE packet • Dumping SAFE Manifest in XML
SAFE I/O – Configuration Status • Configuration identifiers: • safe-1-0-release : an official release of version 1.0 • safe-1-0-rc2 : second release candidate of version 1.0 • safe-1-0-rc1 : a release candidate of version 1.0 • safe-1-0-beta-1: unconsolidated distribution of version 1.0 • Package identifier: esa.safe
XFDU – Environment Information • What is teh exact definition of the environmentInfo/@specVersion attribute? • May the environmentInfo contain package history (e.g. SAFE processing logs)?
XFDU – Information Package Map • What is the presice role of contentUnit/@order ? • GAEL understanding: user defined order • CNES understanding: package sequence number? • A Map section ?shall? Reference all metadata/data objects of the package? • Meaning of multiple Map sections?
XFDU – Single file packages • For DIPs ESA requests single file products • ZIP, Jar or other standard archive format are not suitable for large data sets (e.g. 5 Gb for ENVISAT-ASAR format) • ESA proposes to append binary data objects to the MANIFEST file (not fully compliant with XML 1.0 or XML 1.0)
XFDU – How to design restrictions? • The XML Schema requires global elements and attributes for restricting types in a different target namespace (E.g. SAFE) • May xfdu.xsd propose more global element for that purpose? • Should implementers redefine the xfdu.xsd? • May the XFDU specification provide a clarification/recommendation on that subject?
XFDU – Metadata/Data Object references • metadataObject/@mdRef =? bytestream/@FLocat • metadataObject/@mdWrap =? bytestream/@FContent • May these differences be homogeneized?
XFDU – Checksums • Should the CRC moved or add to the Flocat element (i.e. one CRC per file)? • What is the difference between dataObject/@checksum and bytestream/@checksum?
XFDU – Reference to XLink • Requires an xlink.xsd in the same directory as the xfdu.xsd • References • http://www.w3.org/TR/XLink ? • http://www.w3.org/1999/XLink ? • Where to download the right XML Schema?
XFDU - Typos • vocabluaryMdType ? • Comments of all items of the enumerations: e.g.EAD? • mdSecType may be renamed metadataSectionType?