Earth Observation Payload Data Long Term Archiving The ESA Multi-Mission Facility Infrastructure

1. Earth Observation Payload Data Long Term ArchivingThe ESA Multi-Mission Facility Infrastructure Gian Maria Pinna ESA Eberhard Mikusch, Manfred Bollner DLR Bernard Pruin Werum Software & Systems AG PV-2005 21-23 November 2005

2. MultiMission Facility Infrastructure (MMFI) Rationale • In 2003 the Agency approved a strategy for the evolution of the several missions ground segments (handled and/or to be developed) into an open multi-mission architecture, in accordance with the Oxygen concept for harmonized ESA E.O. Ground Segment implementation • Basic implementation principles: • Adoption of a common architecture for all missions for generic ground segment, mission independent • Decomposition of the facility architecture into functional block elements • Identification of mission specific and common elements • Harmonization and standardization of interfaces • Maximum re-use of already proven elements • Standardization of products and formats across missions • Adoption of strategy for all future ESA handled missions • Extension of concept to European National missions • Harmonization and rationalization of archives

3. Mission A Mission B Mission C Monitoring & Control Mission D Archives User Services I/F Data Management Products Packaging Networks Payload Data Ground Segment Decomposition Specific-to-mission elements(Processors, Acquisition, Q/C, etc.) Examples of multimission common elements

4. Payload Data Ground Segment Logical Model (OAIS-based) Queries Data & orders Interactive User Services Metadata Orders & & browses metadata Data Consumers Data Producers Output Products products Archived products PDGS Data Management PDGS Consumer Access PDGS Administration PDGS Ingestion PDGS Order Processing PDGS Storage

5. A PDGS for a generic mission is composed of: a Multi‑Mission Central Infrastructure component, consisting of all elements required to provide User Services (cataloguing, user access, data ordering, etc.), and Quality Assurance services (payload data quality control, sensor performance assessment, etc.) a distributed Multi‑Mission Facility Ground Segment (FGS) component, consisting of all elements necessary for the acquisition, ingestion, long-term archive, order processing and data disseminations to end users. FGS FGS Producer & Product Services Services Producer & Product Registration Registration Data Data Multi-Mission Central Infrastructure Acquisition Acquisition Ingestion Ingestion Long Term Archive Long Term Archive Order Processing Order Processing Data Dissemination Data Dissemination Data Management Data Circulation User Access QA PDGS Services Model

6. The project FEOMI (Facility Evolution into an Open Multimission Infrastructure) aims at porting the ESA’s FGSs into an MMFI-based architecture, by: Analyzing the present existing operational requirements Consolidating the MMFI architecture in order to satisfy all identified operational requirements Implementing the specific configuration in the Core MMFI to support all operational missions Developing new or Adopting/Adapting existing elements to build the new MMFI, in line with the basic concepts and logical model Developing a generic infrastructure for the usage of the MMFI by future missions’ FGS FEOMI

7. FGS Logical Model Definition • The FEOMI Requirements Analysis phase highlighted the need for: • explicit interoperability model elements for data and metadata exchange, to take care of the dynamics of the distributed archive with federation and co-operation between sites for data exchange, and the synchronization with the central catalogue for metadata and browse images • the mapping of the model elements onto components that were either already in operations in the ESA FGSs, could be created by configuration of COTS or required to be custom built

8. Facility Ground Segment Logical Model Other FGS Multi Multi - - Mission Central Infrastructure Mission Central Infrastructure Monitoring Production Production Exchange Exchange & Control Request Handling Request Handling ISIP IDIP DI DI DI DI Cataloging DI DI Ingest Access Dissemination IDIP SIP DIP Archiving AIP AIP ISIP IDIP Processing IDIP SIP SIP Facility Ground Segment Legend: Legend: AIP SIP SIP – – Submission Information Package Submission Information Package – – Archival Information Package ISIP ISIP – – Internal Submission Information Package Internal Submission Information Package IDIP – – Internal Dissemination Information Package DI DI – – Descriptive Information Descriptive Information DI DIP – – Dissemination Information Package

9. FGS Integration • Following the logical model defined by the FEOMI project, the elements required to build the Facility Ground Segment for a specific mission are: • The so-called Core MMFI, i.e. a set of unconfigured multimission elements and services deployed in a suitable infrastructure. • Several optional Mission Specific Elements (MSEs), accounting for the specificities of the missions sensors data, as seen before typically processing systems. • The specific configuration of the MMFI Elements to allocate the specific services required by the mission, e.g. the ingestion and cataloguing of the specific datasets generated in the FGS.

10. MM Central Services FGS Data Consumers Data Producers MMFI configurations for missions Mission-Specific Elements Facility Ground Segment Decomposition

11. Multi-Mission Central Infrastructure MMMC MMMC MMOHS Data Library Request Handling ULS Local Inventory CAR IF POH POH AMS PR IF Online Archive PSM PSM PSM Processing Processing Processing Product Product Product Processing Processing Processing PSM PSM PSM GFE GFE GFE PFD PFD System System System Processing Processing Processing Distributor Distributor Distributor System System System System System System Other Other IPF IPF Other Other Proc Proc . . IPF IPF Other Other Proc Proc . . IPF IPF Proc Proc . . NRT DDS, ... Site Cache Circulation Circulation Cache In Cache Out Processing Ingestion Dissemination Monitoring and Control Monitoring & Alarm Monitoring & Alarm Operating Tool Logging MMFI Architecture

12. Controlled by the "Generic Front End" (GFE) Configurable workflow engine Registration of data producers and product types Ingestion of products with validation, metadata extraction, browse generation, etc. Generation of master catalogue (MMMC) update files Standard set of re-usable plug-ins relevant to ingestion Standard interface for the implementation of new plug-ins Access to data (including binary data) via Data Request Server (DRS) Metadata extraction and browse generation during ingestion based on the DRS MMFI Ingestion

13. Based on the "Archive Management System" (AMS) and the "Local Inventory" (LI) The AMS manages the actual long-term archiving of the data products Abstraction layer toward underlying storage technology unique storage technology adopted by ESA for its EO missions, in order to achieve maximum harmonization and standardization Automation of operations On-line access via client-server services Internal storage organization transparent to clients Clients data access rights management Products subsetting to handle HBR data with EAST and DRB The LI is the local catalogue indexing all products archived in the long term archive at the centre Indexing and management of on-line archive UpLoad Server (ULS) for metadata synchronization with MMMC MMFI Data Library

14. Based on the “Product Order Handling” (POH) system Handles the on-demand production and dissemination requests received from the ESA’s Multi-Mission Order Handling System (MMOHS) Organizes the required workflow based on the product type and output medium requested in the order Scheduling of data validation and retrieval from Data Library, processing, QC verification, dissemination to users Reports to MMOHS the Production Request statuses The POH is supported by a set of auxiliary components that interface other MMFI elements and provide specific functionality for workflow management MMFI Requests Handling

15. “Product Formatting and Delivery” (PFD) Dissemination workflow management handling different dissemination channels in a configurable manner Optional reformatting and compression (different methods available, including wavelet/JPEG2000) Formatters with specific or generic plug-ins (DRB available for enhanced flexibility) Concurrent dissemination orders with priorities handling Media generation, on tape drives or CD/DVD. Small tape libraries for tapes and RImage CD/DVD Producer Generation of unique medium id with barcode printout, back inlay and delivery note for the end user PFD Network Server addition to perform electronic delivery with dynamic generation of random account and notification to user by e-mail. Notification back to POHMMOHS of URL for user. MMFI Dissemination

16. “Product Distributor“ (PD) Based on PSM (Processing System Management) workflow engine systematic dissemination management (subscription and standing requests handling) systematic product dissemination (by timers and triggers) dissemination via PFD dissemination via DDS and other satellite multicast systems direct dissemination to ftp accounts and file locations product circulation to ftp accounts and file locations product circulation management with state based circulation control dissemination and circulation reporting MMFI Dissemination (2)

17. PSM (Processing System Management) Main function is the abstraction of the processing systems to the higher level elements of the MMFI Allows integrating mission and sensor specific processing facilities with minimal effort and offering a choice of protocols for the definition of the MMFI - processing facility interface Also used in other elements thanks to the workflow model and interface with LI, able to perform complex scheduling of other elements: PSM-CAR (Check And Release) PSM-PR (Product Retrieval) PSM-PD (Product Distributor) Powerful subscription mechanism to LI, based on OQL, to be notified of data appearance in the archive (systematic processing, reprocessing, circulation, etc.) MMFI Processing

18. gfe : GFE drs : DRS li : LI ams : AMS ps : PSM_B ips : Processor mmmc : MMMC 1 : subscribe ( ) 2 : putItem ( x ) 3 : notify ( ) Preceding 4 : putProcessingRequest ( ) interaction suppressed 5 : getItem ( ) 6 : productRetrieval ( ) 7 : start ( ) 8 : done 9 : smartPolling ( ) 10 : productInspection ( ) 11 : browseGeneration ( ) 12 : metadataExtraction ( ) 13 : archiveKey ( ) 14 : productArchiving ( ) 15 : putItem ( register ) 16 : exchange ( add ) Systematic data-driven reprocessing

19. gfe1 : GFE li1 : LI ams1 : AMS pd1 : PD gfe2 : GFE li2 : LI ams2 : AMS 1 : subscribe ( ) Preceding interaction o1 : subscribe ( ) suppressed 2 : putItem ( x ) 3 : notify ( ) 4 : putCirculationRequest ( ) 5 : getItem ( ) 6 : productRetrieval ( ) 7 : create() pr : ISIP 8 : smartPolling ( ) 9 : ingest 10 : productArchiving ( ) 11 : putItem ( register ) o2 : notify ( ) o3 : circulationAcknowledge ( ) o4 : destroyItem ( ) MMFI Use Cases - Circulation

20. The MMFI implements various features covering preservation and value adding concepts: Support to operational scenarii for preservation strategies, e.g. periodic migration of digital products to new information technology Encapsulation by self-describing items as defined by OAIS model information packages. The maintenance of metadata is performed and means for data consistency are supplied Support for automated production by means of sophisticated data access and processing management Modular design, open architecture and streamlined interfaces that permit an easier substitution of one or more of its elements, if the need will arise in the future, to ensure the long‑term preservation of its data holdings and of its services Special attention was put on the architecture for a well balanced assignment of functionality-to-components. MMFI Advanced Features

21. Archiving technology migration, by shielding the physical data-sets from the applications, using several software layers, that can all be used to handle lower level technology changes: Hierarchical Storage Management (HSM) incorporated in the AMS. A potential change of the HSM technology would be fully resolved in the AMS. Limited number of components directly interfacing the AMS, thus also a change of the AMS or AMS interfaces could be handled with minimal archive impact. Technological evolution and scalability through the modularity, achieved by the architecture largely built from autonomous, networked components that can be combined by configuration: Simplified exchange of some components by other implementations Handling of increased load by instantiating more components in optimised configurations (e.g. increase the number of processing nodes) Processor integration, to cope with changes to the Data Processors used for adding value to the EO data: The PSM framework supports the integration of processor by natively supporting a variety of processor interfaces and by allowing integrating processor adapters The flexibility of this approach makes it possible to substitute processors and processor interfaces without undue effort and without affecting other parts of the participating workflows Product data model migration, to cope with the evolution of processing algorithms that requires changes in the data models of the products to be archived: Configurable product object models within the Data Library that can be extended MMFI System Design Features

22. MMFI Functional Features

23. Data Library Local Inventory AMS Level x Notification V1.0, V2.0, (V1.0+2.0) Level 0 Notification Product Distributor PSM Acq. Processing Processing System PSM Other Proc. IPF Processing Dissemination User Access Online Archive Re-Processing Pointer Processing System User PSM CirculationCache Out PFD Other Proc. IPF Re-Processing V 2.0 V 1.0 Level 0 Level x MMFI Systematic Workflows