1 / 21

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps. Ruth Duerr, NSIDC Christopher Lynnes, GES DISC Mike Folk, Kent Yang, Peter Cao, The HDF Group. HDF4 files are complex. How do we save HDF users from having to deal with all of the complexity under the hood?.

emele
Download Presentation

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Ruth Duerr, NSIDC Christopher Lynnes, GES DISC Mike Folk, Kent Yang, Peter Cao, The HDF Group HDF/HDF-EOS Workshop XIII

  2. HDF4 files are complex HDF/HDF-EOS Workshop XIII

  3. How do we save HDF users from having to deal with all of the complexity under the hood? HDF/HDF-EOS Workshop XIII

  4. Through the HDF software libraries, either by using the HDF APIs directly or by using HDF tools that depend on the HDF libraries. But what about the future… HDF/HDF-EOS Workshop XIII

  5. There is a risk in depending solely on HDF libraries to access HDF-formatted data over the long term. It is possible, especially in the distant future, that the libraries may not be available. HDF/HDF-EOS Workshop XIII

  6. “If only we could read HDF data with an independent program that does not rely on the HDF API… A possible approach [would be to create] a map of a data file, [and] utilities to find, assemble and write out SDSes and vdatas.” “Leveraging HDF Utilities”Christopher LynnesHDF Workshop X. HDF/HDF-EOS Workshop XIII

  7. User’s view of the HDF4 SD model HDF/HDF-EOS Workshop XIII

  8. Mapping SDS to file offset/length HDF4 file layout HDF/HDF-EOS Workshop XIII

  9. Mapping with chunks HDF4 file layout HDF/HDF-EOS Workshop XIII

  10. Recap • Problem • The complex byte layout of HDF files makes long-term readability of HDF data dependent on long-term availability HDF software. • Solution • Create a map of the layout of data objects in an HDF file, allowing a simple reader to be written to access the data. HDF/HDF-EOS Workshop XIII

  11. The project – phase 12007-2008 HDF/HDF-EOS Workshop XIII

  12. HDF4 mapping project activities • Assess and categorize HDF4 data held by NASA • Determine what types of objects to map. • Get an idea of the magnitude of the project. • Develop prototype for proof of concept • Develop markup-language based layout specification. • Develop tool to produce layout for an HDF4 file. • Develop and test two independent tools to read HDF4 data based solely on the map files HDF/HDF-EOS Workshop XIII

  13. How many HDF4 products? HDF/HDF-EOS Workshop XIII

  14. Product Identification Product Name Data Level Archive Location Product Version Whether the product was multi-file For HDF-EOS products HDF-EOS version For point data Number of point data sets Maximum number of levels For swath data Number of swaths Maximum number of dimensions Organized by time, space, both, or other Whether dimension maps were used For gridded data Number of grids Max number of dimensions in a grid Number of projections used Whether any grids were indexed HDF Version For raster data Number of 8-bit rasters Number of 24-bit rasters Number of general rasters Whether any rasters had attributes Whether any rasters were compressed Whether any rasters were chunked Whether there were any palettes For SDS data Number of SDSs Maximum number of dimensions Did any SDS have attributes Was any SDS annotated Were dimension scales used Was compression used and if so what kind Was chunking used For Vdata Number of Vdata structures Did any Vdata have attributes Did any Vdata fields have attributes Was compression used and if so what kind Was chunking used Data characteristics Product Characteristics Examined HDF/HDF-EOS Workshop XIII

  15. HDF4 mapping prototype workflow HDF4 File “H4.hdf” HDF4 Mapping File (XML document) “H4.hdf.map.xml” hmap linked with HDF4 library Object Data Groups, Data Objects, Structural and Application Metadata; Locations of Object Data Reader 1 (C program) Reader 2 (Perl Script) HDF/HDF-EOS Workshop XIII

  16. Phase 2: 2009-2011Productizing HDF4 Mapping schema and tools for deployment HDF/HDF-EOS Workshop XIII

  17. Phase 2 tasks • Revise schema • Investigate integration of mapping schema with existing standards • Analyze what’s needed to include HDF-EOS 2 • Revise the XML schema • Implement production quality HDF4 map writer • Develop demo HDF4 map reader • Deploy • Optional tasks • Implement general purpose reader • Develop validation utilities HDF/HDF-EOS Workshop XIII

  18. How you can help • Project page at The HDF Group website: • http://www.hdfgroup.org/projects/hdf4mapping/ • Consider what it might take to implement this for your archive - contact us if you’d like support • Let us know if you are interested in participating in any capacity. HDF/HDF-EOS Workshop XIII

  19. Thank You! HDF/HDF-EOS Workshop XIII

  20. Acknowledgements This work was supported by cooperative agreement number NNX08AO77A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. HDF/HDF-EOS Workshop XIII

  21. Questions/comments? HDF/HDF-EOS Workshop XIII

More Related