Feasibility Study

1. AVHRR 1 km Data Set ConsolidationConsolidating 30 years of EO data hosted by multiple data holders 28th LTDP WG Meeting Frascati (ESA/ESRIN) 04–05 November 2014

2. Feasibility Study Inter-agencyactivity - Consolidateandmakeavailable AVHRR 1 km (LAC) dataheldbydatacenters in Europe and Canada Feasibility Study:

3. User Requirements Capture Survey distributed to selected users to capture user scenarios and user requirements for • Data and metadata – format, structure, content • Preservation and usability - data records, information & tools • Discovery and retrieval • Exploitation tools & infreastructure

4. User Requirements Capture Respondents • Brockmann Consult, Germany • Norwegian Meteorological Institute, Norway • University ofBerne, Switzerland • EidgenössischeTechnischeHochschule Zurich, Switzerland • Helmholtz-ZentrumGeesthacht, Germany • Remote Sensing Centre of Institute of Geodesy and Cartography, Warsaw, Poland • DLR, Germany • ESA CCI

5. User Requirements – Summary 1 Data • Level L0 to L2 for user access • Standard formats: NetCDF, HRPT, HDF, ASCII, ESA Beam compatible • Consistent across satellite generations, quality flags Metadata • Product metadata: OGC-EOP • Following CF (Climate and Forecast) standard • Keep inside in the hdf5/netcdf4 • Collection metadata: ISO 19115 and INSPIRE compliant • Controlledvolcabularies

6. User Requirements – Summary 2 Preservation and usability Data records • Preserve HRPT raw & auxiliary data • Reprocess as calibration data or method changes • Keep some (tbd.) generations of the previous data sets • Complete datasets • Consistent versioning / numbering scheme • Referenced by DOI Associated knowledge • Information: NOAA docs, provenance, quality - available to user • Tools: processing chain, visualization, and analysis • HW & OS independent, maintain libraries, open source preferred • Test suite

7. User Requirements – Summary 3 Discovery • Centralized online portal with advanced browse capability • Searchable by: satellite name, time frame, spatial domain, orbit number, product version • Standardized interfaces, machine-harvestable catalog (OAI-PMH, OGC CSW) Retrieval • Same data format and access interfaces/protocols if data held in distributed achives • Automated download using data retrieval script • Request for data streams instead of download - OpENDAP preferred • Time series retrieval of stacked products for a single pixel (e.g. ASCII) • Direct download of bulk data • Data ordering not favored • Distribution option physical media

8. User Requirements – Summary 4 Exploitation tools & infrastructure • Tools: processing chain, visualization, and analysis • Exploration and exploitation capabilities provided by ESA desirable • Possibility to integrate other data sources beyond EO • May not be necessary if OPeNDAP access is provided

9. Summary • There is a demand for a complete, consistent, calibrated time series • Good agreement on data format: NetCDF, CF • Diverse requirements for processing level • Central discovery & access • Procedures, provenance, etc. must be documented and available

10. Questions? • Tyler Christensen • DLR - German Remote Sensing Data Center • Information Technology • tyler.christensen@dlr.de

11. Backup Slides

12. Analysis DLR and Dundee Inventories Number of NOAA AVHRR 1 km scenes (passes) available at DLR and at Dundee Satellite Receiving Station. Duplicate scenes have been removed for the analysis. The purple line indicates the approximate number of scenes required to ensure an adequate spatial and temporal coverage for the AOI of the TIMELINE project (6 passes x 24 hrs x 365 days). Number of scenes (orbits) Year AOI of the DLR TIMELINE project

13. Details of User Requirements

14. User Requirements - 1 User Scenarios • Users will process raw data into their own interpreted products: land cover, snow extent, water body extent, NDVI, fires and burned area, etc. • Some users want higher-level AVHRR products (e.g. snow, surface temperature) that they can compare to their own model outputs • Consolidated data set would be used to extend existing analyses in space and/or time, sometimes using existing processing chains—so data format will be very important

15. User Requirements - 2 General Requirements for Consolidated Dataset • All levels requested, from L0 all the way to L2 interpreted products • Some users need data as raw as possible—no atmospheric correction, individual swaths not stitched, not reprojected, etc. • Some users need fully processed data—mosaics, georeferenced, time-aggregated (daily) • Must include quality flags & pixel-center location grids • Must be internally consistent (i.e. corrections applied between different satellites for a smooth time series), but should be fully documented • Ancillary data: ozone, water vapor, aerosol properties if available

16. User Requirements - 3 Data - format, structure, content Format • Most data users requested NetCDF, one said HDF would also be OK. • A few also required the raw and L1b data in original HRPT format. • Perhaps also ASCII, especially for a time-series of an individual pixel. • Readable with ESA BEAM toolbox. Structure • Descriptive, searchable file names—sensor, dates, mission, etc. • NetCDF files should have metadata headers—some requested that metadata follow the Climate & Forecast (CF) convention. Content • Separate scientific data sets (SDS) for reflectances, brightness temperatures, acquisition angles, scanline times, and NWP data used for atmospheric correction. • Requested in metadata header: long_names, grid information, standard_name

17. User Requirements - 4 Metadata (collection, product) - format, structure, content Format • CF (Climate and Forecast standard) metadata inside NetCDF file • Collection metadata: ISO 19115 and INSPIRE compliant • Product metadata: OGC-EOP • Keep inside in the hdf5/netcdf4 Structure • NetCDF Discovery Attribute Convention Content • Follow CF-conventions for netCDF data • Utilize controlled vocabularies (C&F standard names, GCMD Science Keywords) • Sensor, platform, temporal and spatial coverage, calibration information, applied processing chain, ancillary data and parameters used

18. User Requirements - 5 Preservation and Usability Needs • Reprocess dataset whenever the calibration data or the calibration method changes • Complete datasets • Any changes to the dataset need to be carefully documented and the information needs to be made available to the user – via the metadata or an easily and permanently accessible document. • Datasets should be referenced by DOI • A consistent versioning / numbering scheme is required • Some (tbd.) generations of the previous data sets should be kept

19. User Requirements - 6 Data Record Needs • Original HRPT raw & auxiliary data must be kept internally, to allow for future reprocessing as improved algorithms, new data formats, or different intercalibrations emerge. • Metadata and browse images should also be kept. • L1b data & L2 products– possibly several generations of re-processed data • Must be kept in standardized data formats

20. User Requirements - 7 Additional Information and Tools to be Preserved • NOAA documentation (e.g. POES user guide) should be available and linked to data. • Data processing documentation should be kept: procedures for aggregation, temporal averaging, inter-calibration, etc. • Information on provenance should be archived: platforms, sensor characteristics of each platform, calibration updates etc. • If possible all software required to generate each version of the product should be maintained: calibration coefficients, NWP data, tle information, etc. • Quality assessments, validation and inter-comparison reports • Data use policies • Data format descriptions

21. User Requirements - 8 Needs for Software and Tools • Software content to preserve: processing chain software, visualization tools, data readers for HRPT and other formats, and BEAM tools (i.e. data readers, visualization, processing, analysis) • Also keep copies of libraries that the software uses, and/or compile the software against static libraries so it would be usable even if newer library versions are not compatible. • Try not to depend on specific hardware or OS; use standard software platforms and compilers. • Need a test suite. • Open source preferred.

22. User Requirements - 9 Discovery Needs • Centralized online portal with quick looks and advanced browse capability, searchable by: satellite name, time frame, spatial domain, orbit number, product version • A catalogue interface with machine-harvestable metadata would be useful. One user states that OAI-PMH is preferred over OGC CSW due to the simplicity in the interface. • Should be no access difference between historic and current missions • Data policies should be aligned

23. User Requirements - 9 Data Retrieval Needs 1 • Automated download using a data retrieval script should be possible • Data sets can be hosted in separate archives, however, the same data format and access interfaces/protocols (FTP, WCS, other?) should be used by all providers • Request for data streams which users could attach a processing chain to, instead of downloading a local copy of the dataset • OpENDAP access is preferred. • Direct download of bulk data using shell scripts, query specifying file name pattern, etc. Preference for selectable times/locations and 'mission-at-once'.

24. User Requirements - 10 Data Retrieval Needs 2 • Standardized formats and interfaces/transfer protocols in case data need to be transferred to third party processing infrastructure • Retrieve a time series of stacked products for a single pixel. • Data ordering is acceptable but not preferred. • Distribution on physical media (as opposed to online download) is also requested.

25. User Requirements - 11 Exploitation Infrastructure Needs (hosted processing) • Most users don’t think this is necessary, especially if OPeNDAP access is provided. • One user requested exploration and exploitation capabilities provided by ESA, with a possibility to integrate with other data sources beyond EO. Would require a collaborative effort to build virtual data exploitation platforms. • Exploration and exploitation capabilities provided by ESA desirable

26. User Requirements - 12 Needs for Current (vs. historic) AVHRR Data • New data are as important as the historical archive. • Most users update their products regularly, and would like to have daily updates as new data become available. • Some users do not need updated data, if they are doing retrospective analysis. • New data should have exactly the same format as archived data.

27. User Requirements - 13 Time Series Needs • Most users will download the data to a local copy, and then perform their own time series analysis. • Request to receive a time series for a single pixel as a text file. • Remote data processing and/or OpENDAP access would reduce the need for users to download large amounts of data for the whole time series.