1 / 11

Leveraging HDF Utilities

Leveraging HDF Utilities. Christopher Lynnes Goddard Earth Sciences Data and Information Services Center (GES DISC). History Began as part of EOSDIS Version 0 (1992-1994) with UARS, TOMS and TOVS Now handles data from TRMM, Aqua and Aura satellites Production, Archive and Distribution

mhicklin
Download Presentation

Leveraging HDF Utilities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Leveraging HDF Utilities Christopher Lynnes Goddard Earth Sciences Data and Information Services Center (GES DISC)

  2. History Began as part of EOSDIS Version 0 (1992-1994) with UARS, TOMS and TOVS Now handles data from TRMM, Aqua and Aura satellites Production, Archive and Distribution Value-added Services Formats: HDF4, HDF-EOS HDF5, HDF-EOS5 GRIB, BUFR, other GES DISC

  3. Pre-cut subsets (run as part of production) Create software to write HDF files On-the-fly subsets (run during FTP transfer) OPeNDAP servers OpenGIS access Giovanni: Server-side Visualization and Analysis Create software to read HDF files Value-Added Services

  4. Online Analysis: • Maps • Time-Series • Correlations • ASCII Output

  5. Giovanni Output Example

  6. Command-line utilities are easy to integrate Ncdump  S4PM Production System Reads metadata from HDF-EOS global attribute Hdiff  Science Software Integration and Test process Compare output with that from algorithm developer Hrepack Data production software Data Pool “compression-on-insert” “In the Beginning was the Command Line…”1 1Neil Stephenson

  7. Before hrepack: Internal compression was little-used Have to write the whole SDS at once External compression more difficult to manage Use-in-place, file metamorphosis… Then… Szip test driver1 (2002) “hdfzip” proof of concept (2003) Reimplementation by NCSA as hrepack (2003) Now: Allows us to archive 2-4 times as much data online Critical enabler of EOSDIS Evolution The Impact of hrepack 1 Wei Xia-Serafino

  8. HDF API Significant spin-up time for new data users Command-line lowers the entry threshold Ncdump, hdp used to extract to binary or ascii BUT: Would be easier with easy-to-find static binaries packaged as standalone utilities Download-and-use right away No configure/make trip-ups Lowering the Learning Curve

  9. Users want HDF reformatted to: netCDF GeoTIFF GRIB ASCII GES DISC would like the reverse (X  HDF) for Giovanni Next: Reformatters?

  10. Solve only for data structures/conventions that are tractable? HDF-EOS, COARDS, CF-1 Provide extra information to map formats? Semantic maps Data structure maps Earth Science Markup Language Attacking the Reformatting Problem

  11. If only we could read HDF data with an independent program that does not rely on the HDF API… HDF as an archive format HDF as a NASA data system standard Possible approach: “Extend” hdfls to print a hierarchical map of a data file Write ncdump/hdp-like utilities to find, assemble and write out SDSes and vdatas Speaking of Mapping Formats…

More Related