1 / 35

Vincenzo Beruti, Mirko Albani European Space Agency DPIF Symposium, April 21 -23, 2010

The European Space Agency Earth Observation Long Term Data Preservation programme and experiences across CASPAR, GENESI-DR and PARSE projects. Vincenzo Beruti, Mirko Albani European Space Agency DPIF Symposium, April 21 -23, 2010 Dresden, Germany. Outline.

onawa
Download Presentation

Vincenzo Beruti, Mirko Albani European Space Agency DPIF Symposium, April 21 -23, 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The European Space Agency Earth Observation Long Term Data Preservation programme and experiences across CASPAR, GENESI-DR and PARSE projects Vincenzo Beruti, Mirko Albani European Space Agency DPIF Symposium, April 21 -23, 2010 Dresden, Germany DPIF Symposium 2010, Dresden, Germany

  2. Outline • Earth Observation data preservation: the starting point and the needs • The EO European challenge and the ESA EO LTDP strategy • The European EO LTDP Framework • The activities performed • The concept and the guidelines • The ESA EO LTDP programme (2009-2011) • The ongoing activities • The next steps • The experience across EC projects like Caspar, Genesis-DR, Parse • Conclusion DPIF Symposium 2010, Dresden,21-24 April 2010

  3. Outline • Earth Observation data preservation: the starting point and the needs • The EO European challenge and the ESA EO LTDP strategy • The European EO LTDP Framework • The activities performed • The concept • The Guidelines • The ESA EO LTDP programme (2009-2011) • The planned activities • The Next steps • The experience across EC projects like Caspar, Genesis-DR, Parse • Conclusion DPIF Symposium 2010, Dresden,21-24 April 2010

  4. Earth Observation data preservation: the starting point • The preservation of EO space data has to be considered as a long term responsibility and a challenge of the Space Agencies or data owners as they constitute a humankind asset. • Preservation includes data integrity, data awareness, data access and exploitation • ESA’s EO archives • Content of EO data archives is extending from a few years to several decades • Very valuable data representing scientific long time-series for a large number of applications • ~150 TB archived in early ‘90s, about 3 PB archived today and over ~30 PB expected in the next 10 years • ESA EO archives are located in 12 locations in 9 European countries (distributed archives) • Same increasing trend is expected in other European EO Satellites owners archives DPIF Symposium 2010, Dresden,21-24 April 2010

  5. LTDP Earth Science Applications • In order to satisfy new requirements, the Earth Science community increasingly need access to new missions data and to historical datasets, spanning 30 years and more • Application examples • monitor sea-level & surface temperatures increase, observe seasonal growth of plankton, gauge ocean winds & currents, detect variations in thickness of ice sheets • detect changes in land surface, monitor vegetation growth, measure surface temperatures & soil moisture • watch out for symptoms of climate change in ice cover • changes in cloud cover and other atmospheric features DPIF Symposium 2010, Dresden,21-24 April 2010

  6. EO data application example DPIF Symposium 2010, Dresden,21-24 April 2010

  7. Outline • Earth Observation data preservation: the starting point and the needs • The EO European challenge and the ESA LTDP strategy • The European EO LTDP Framework • The activities performed • The concept and the guidelines • The ESA EO LTDP programme (2009-2011) • The ongoing activities • The Next steps • The experience across EC projects like Caspar, Genesis-DR, Parse • Conclusion DPIF Symposium 2010, Dresden,21-24 April 2010

  8. The EO European challenge • Traditionally in Europe there has been poor cooperation in this field with no common approach for long term preservation and access to EO space data. • Single organizations have difficulties to afford data preservation in the long term that calls for the need of optimising costs and efforts, identifying commonalities. • A coordinated and coherent approach (i.e. a European EO LTDP Framework) for a harmonized management of European archives is needed to guarantee the European EO data set preservation. • ESA, within its mandate for space coordination coordinates and shares a Common Long Term Data Preservation approach among all the stakeholders. DPIF Symposium 2010, Dresden,21-24 April 2010

  9. The ESA EO long term data preservation strategy: phase 1 • Phase 1 • 2009-2011 ESA LTDP funded programme • Enhancement of ESA EO archiving facilities infrastructure • Technology studies, data set definition, archives interoperability, certification, security, etc. • Standardization issues • European and International Organization interfaces/projects (like Caspar, Parse, Genesis-DR) • European EO LTDP framework coordination • Common guidelines definition • Detailed LTDP framework Workplan completion • Program proposal for Phase 2 , including funding scheme DPIF Symposium 2010, Dresden,21-24 April 2010

  10. The ESA EO long term data preservation strategy: phase 2 • Phase 2 • 2012-2018: European LTDP Framework • Collaborative framework open to all possible members consisting of distributed and heterogeneous components and entities cooperating in several areas to reach an harmonized preservation of the European EO Space Data Set • Sustained through a cooperative long term and programmatic funding framework • Cooperation areas /activities • Policies • Technology, methodology and developments • Standardisation activities in close link with international bodies (CCSDS, CEOS, OGC, INSPIRE, EU initiatives) • Operational solutions • Data exploitation DPIF Symposium 2010, Dresden,21-24 April 2010

  11. Outline • Earth Observation data preservation: the starting point and the needs • The EO European challenge and the ESA EO LTDP strategy • The European EO LTDP Framework • The activities performed • The concept and the guidelines • The ESA EO LTDP programme (2009-2011) • The ongoing activities • The Next steps • The experience across EC projects like Caspar, Genesis-DR, Parse • Conclusion DPIF Symposium 2010, Dresden,21-24 April 2010

  12. LTDP EO Framework Goals • Preserve the European* EO space data set for an unlimited time-span • Ensure and facilitate the accessibility and usability of the preserved European* EO space data set. • Respect the individual entities applicable data policies • Implement a cooperative and harmonized collective approach • Possibly ensure the coherency with the preservation of other non-space based environmental data and international policies. *including Canada DPIF Symposium 2010, Dresden,21-24 April 2010

  13. LTDP EO Framework Concept • Collaborative framework with contribution of European EO Space data owners: • consisting of distributed and heterogeneous components and entities • cooperating in several areas to reach an harmonized preservation of the European EO Space Data Set • through their ideas and possibly their infrastructure and in accordance to commonly agreed LTDP Guidelines • Aiming at a multilateral Cooperation with multiple funding sources from at least the European EO data owners • The existence of an European LTDP Framework can trigger the availability in the long term of additional permanent funding to be made available from different sources • Progressive implementation • based on a stepwise work-plan approach (short, mid, long-term activities). • Open to all possible members including the possibility for any European EO data owner to preserve missions data beyond their funding schemes DPIF Symposium 2010, Dresden,21-24 April 2010

  14. What has been done • A leading group of European Agencies owning EO mission archives have joined forces in the LTDP Working Group to address this challenge together since 2006. DPIF Symposium 2010, Dresden,21-24 April 2010

  15. What has been done • 1st European workshop held at ESA/ESRIN in May 2008 aiming at: • Collect and exchange information on LTDP policies and technical approaches. • Collect feed-back for a potential European LTDP common approach and strive deriving common recommendations for it. • Assess the impact (benefits, drawbacks) of the proposed European LTDP Common Guidelines with each archive owner • Consolidated: • The European LTDP 9 Common Guidelines (Data set definition, Archives maintenance and data integrity, Archives operations, Data security, Data ingestion, Data access and interoperability, Data exploitation and re-processing, Standardization, Data Purging/Appraisal • The European LTDP Framework concept and the implementation plan • Initiated a number of technical activities related to Long-term data preservation. • An LTDP programme has been approved at ESA with limited funds available for the period 2009-2011. DPIF Symposium 2010, Dresden,21-24 April 2010

  16. LTDP programme approved at ESA for the period 2009-2011. Outline • Earth Observation data preservation: the starting point and the needs • The EO European challenge and the ESA EO LTDP strategy (period 2007-2009) • The European EO LTDP Framework • The activities performed • The concept and the Guidelines • The ESA EO LTDP programme (2009-2011) • The ongoing activities • The Next steps • The experience across EC projects like Caspar, Genesis-DR, Parse • Conclusion DPIF Symposium 2010, Dresden,21-24 April 2010

  17. ESA EO LTDP Programme Workplan 2009-2011 LTDP programme approved at ESA for the period 2009-2011. LTDP programme approved at ESA for the period 2009-2011. DPIF Symposium 2010, Dresden,21-24 April 2010

  18. Ongoing Activities • Consolidation of user requirements, including those from the Climate Change Initiative and GMES • Study in cooperation with other European EO data archive owners in order to define the composition of the “data set” to be archived and to guarantee knowledge preservation (FIRST) • Technological study (LAST) covering the collection and analysis of the operational requirements and an archiving technology survey in short, mid and long term and bench marking/prototyping activities on the recommended technologies • An LTDP Programme Management support contract is set up aiming at the procurement of support for the preparation of additional studies and activities in the field of Archive security, archive certification, networking, etc., including support to the preparation of the programmatic approach for the period above DPIF Symposium 2010, Dresden,21-24 April 2010

  19. Ongoing Activities • ESA has upgraded the Tape libraries at most of the EO archive centers, replacing the STK Powerhorn Tape Libraries based on 9940 B technology with the new SUN SL3000 based on T10000 B based technology. • Other initiatives will be soon initiated in order to: • recover data from historical missions • enhance the on line archive data access • implement faster data reprocessing • explore adoption of archive virtualization for data distribution • adopt new metadata search methodologies including data mining techniques • Enhance security of archives and implement archives certification • Enhance archives inter-operabilty with the adoption of defined standards DPIF Symposium 2010, Dresden,21-24 April 2010

  20. Next Steps • LTDP WG activities will continue, with the involvement of all European data owners and archive holders, to coordinate and prepare the set-up of the European LTDP Framework: • LTDP guidelines consolidation and promotion within CEOS or GEO • Activities for archives security and certification • Archives Interoperability • Data exploitation enhancements, including support to the Climate Change initiative • Analysis of reprocessing infrastructure • Alliance partnership and other EC funded projects support • ESA is already planning to apply the Long Term Data Preservation Common Guidelines to its own missions • High priority projects will be initiated in the next two years at ESA focusing on data preservation and enhancement of data access • The most important activity will be the preparation of the LTDP programme proposal for the period beyond 2011 to be presented to Delegations DPIF Symposium 2010, Dresden,21-24 April 2010

  21. Outline • Earth Observation data preservation: the starting point and the needs • The EO European challenge and the ESA EO LTDP strategy (period 2007-2009) • The European EO LTDP Framework • The activities performed • The concept and the guidelines • The ESA EO LTDP programme (2009-2011) • The ongoing activities • The Next steps • The experience across EC projects like Caspar, Genesis-DR, Parse • Conclusions DPIF Symposium 2010, Dresden,21-24 April 2010

  22. CASPAR Main Outcomes (1) • CASPAR has specified and built components for a framework • based on the OAIS reference model • to support the end-to-end preservation lifecycle for digital information • The CASPAR framework is an open system • able to interoperate with as many different systems as possible • to be operated in the framework of existing preservation solutions and be re-implemented as systems evolve DPIF Symposium 2010, Dresden,21-24 April 2010

  23. CASPAR Main Outcomes (2) • CASPAR put knowledge (not just the “bits”) at the heart of preservation • development of techniques to preserve the information and knowledge Representation Information that is encoded in digital objects (via Semantic) • design and development of high-level knowledge management services for digital information preservation systems (based on Semantic Web technologies) • preservation of authenticity and digital rights • ESA has participated in many tasks with a leading role in Stream 4 (Testbed, Testing and Validation). • 3 testbeds (UNESCO, IRCAM, ESA) to validate CASPAR solutions • CASPAR Scientific Testbed • ESA user and data/infrastructure provider DPIF Symposium 2010, Dresden,21-24 April 2010

  24. CASPAR Benefits • Framework validation (CASPAR components are suitable for preservation of ES data) • Lesson learnt • preservation of knowledge associated to data • preservation not only of data but also of data processing • best practices to cope with long term data preservation problems by using OAIS model real applications • Main results • development of a 100% CASPAR components based framework (ESA CASPAR System is available for further enhancement/testing and for users and data owners/providers willing to see a practical approach to preservation using CASPAR solutions) • demonstration of the suitability of CASPAR solutions for applications in the Earth Science field (and in the ESA EO Ground Segments infrastructure) • integration with GENESI-DR to validate in a more complete form the CASPAR data preservation framework in the Earth Science domain and to promote the “CASPAR preservation model” in a wide community sharing the ESA CASPAR experience with other ES stakeholders DPIF Symposium 2010, Dresden,21-24 April 2010

  25. GENESI-DR: the key points Ground European Network for Earth Science Interoperations –Digital Repositories: an Earth Science e-infrastructure federating Digital Repositories • Discovery of heterogeneous distributed data (in situ, satellite, airborne) from different data providers; • Controlled access to data in the respect of the data providers’ local policies; • Access to high performance processing services; • Strong scalability for easy federation of new Digital Repositories; • Adoption of data curation and preservation solutions for selected federated data. DPIF Symposium 2010, Dresden,21-24 April 2010

  26. Demos HR DEM and aerial photography • Chlorophyll water data • Chlorophyll Terrestrial data • POLDER products • MERCATOR products • SPOT 1- 5 • Landsat 5, 7 • SRTM • Chorographic maps of the Italian territory • Color Orthophotos VOS/Buoys SWACI Infoterra CNES JRC ENEA SSTT DLR NDACC Lidar Min for Env CNR CENTRAL SITE ISPL ASI NILU GENESI-DR Web Portal NASA SIR_C_X_SAR ESA CENIA KSAT OMI NO2 Level 3 • GLOBmodel (ozone columns, temperature) • SCIAMACHY • OMI DOAS O3 column • ENVISAT (MERIS,ASAR,AATSR,SCIAMACHY) • ERS-2 (GOME,SAR) • PROBA (CHRIS,HRC) • DORIS (POR, VOR) • L3 products • CORINE Land Cover 1990, 2000 • High resolution images of the Czech Rep. • ENVISAT ASAR • KOMPSAT2 • Vessel/wind detection The GENESI –DR current deployed topology DPIF Symposium 2010, Dresden,21-24 April 2010

  27. The GENESI-DR Web Portal Users can discover data and services via the GENESI-DR Web portal …geographical area, temporal range.. searching on the base of: DR DR CENTRAL SITE DR DRs reply with all data and services matching the query …free text string …and other specific parameters as applicable DPIF Symposium 2010, Dresden,21-24 April 2010

  28. Data processing in GENESI-DR GENESI-DR provides on demand processing capabilities: application/algorithms are run on Grid resources GENESI-DR splits the processing steps in several jobs. These arerun in parallel as possible in different computing nodes of the underlying Grid infrastructure. Expert users are so enabled to produce the final desired product. DPIF Symposium 2010, Dresden,21-24 April 2010

  29. Data Curation in GENESI-DR • The core activity of the work package dedicated to Data Curation, after a critical analysis of existing standards and solutions, i.e. OAIS Reference Model, CASPAR EC project and INSPIRERecommendation, was aimed at: • Assessing Data Curation solutions (if any) already adopted by each federated DR; • Experimenting the adoption within GENESI-DR of data preservation and curation mechanisms defined in CASPAR; • Defining an interoperable data curation metadata model expanding the GENESI-DR metadata core set; • Specifying a GENESI-DR data curation strategy at both DR and federation level; • Developing a Submission Tool for uploading and ingesting data in the GENESI-DR federation in compliance with the OAIS reference model. DPIF Symposium 2010, Dresden,21-24 April 2010

  30. PARSE In-sight Project • PARSE.Insight includes a case study on EO, represented by ESA • ESA’s partnership and activity in PARSE.Insight is perfectly aligned with its role of coordinator at European level to ensure preservation and accessibility for ESA’s and Member States’ EO data in the long-term • In the context of PARSE.Insight, ESA issued a public user consultation to get an insight about community’s awareness of data preservation and current/envisaged exploitation of historical environmental data streams, including opportunity of experience DPIF Symposium 2010, Dresden,21-24 April 2010

  31. PARSE ESA Objectives • To contribute to fill the gap between the EO data generation and archiving and the EO data exploitation • To understand EO data users’ standpoint and requirements about historical space/non space data exploitation • To provide input to the European strategy for preserving EO data DPIF Symposium 2010, Dresden,21-24 April 2010

  32. Parse Survey main conclusion • The scientific community need and want to access historical environmental data and historical time series of earth observations • The community want to enhance its experiences on historical data exploitation aiming at a more active involvement in the process with reporting examples and suggestions • The EO data users are aware and updated on the current infrastructures’ constraints and require timely solutions • To pursue the implementation of an e-infrastructure • Provide input to the ESA LTDP programme, to enhance the understanding of EO space data to be preserved and how to ensure environmental non space data preservation in parallel for a common exploitation in future applications DPIF Symposium 2010, Dresden,21-24 April 2010

  33. EC funded projects contribution The three projects have contributed to the ESA long term view concerning the issue of Digital Data Preservation in several areas: - Analysis, implementation and validation of a test bed for multi- disciplines Digital data Preservation in accordance to International standards • Operational experience for an operational access to a distributed system of Digital Repositories and utilization of Grid based technologies for data processing of multi-applications in order to extract added value information • Practical implementation and experience of several digital data curation issues • In-sight depth knowledge about EO historical data exploitation users requirements and experience DPIF Symposium 2010, Dresden,21-24 April 2010

  34. Outline • Earth Observation data preservation: the starting point and the needs • The EO European challenge and the ESA EO LTDP strategy (period 2007-2009) • The European EO LTDP Framework • The activities performed • The concept and the Guidelines • The ESA EO LTDP programme (2009-2011) • The ongoing activities • The Next steps • The experience across EC projects like Caspar, Genesis-DR, Parse • Conclusions DPIF Symposium 2010, Dresden,21-24 April 2010

  35. Conclusions • The ESA LTDP Programme is progressing, targeting very clear objectives: • To provide an adequate response to the user needs ensuring data • integrity and access with a very long perspective • To define and implement a European framework involving all • concerned European actors in order to face together the future • challenges, including funding issues • To enhance the ESA infrastructure in order to adequately respond on • a solid ground to all current and future archive technical requirements • To aim at a coordinated world wide approach for increasing the • awareness about EO long tem data preservation • To integrate contributions from different demonstration projects funded by the EC and other sources into a unique vision for current and future digital data preservation DPIF Symposium 2010, Dresden,21-24 April 2010

More Related