HDF-EOS Status, Related Tools and Issues

1. HDF-EOS Status, Related Tools and Issues Abe Taaheri, Raytheon IIS 2009 THG Briefing to ESDIS

2. TOOLKIT / HDF-EOS Support • Maintenance support is funded on EMD Evolution through May 22, 2009 - Bridge contract until EODDIS Evolution & Development (EED) award in 2010 • Current support • HDF-EOS 2 & 5 • SDP/MTD Toolkit • HDF-EOS plug-in for HDFView • HDF-EOS2 to GeoTIFF Converter (HEG) • Other Toolkit/HDF-EOS related software • Toolkit/HDF-EOS/HEG helpdesk

3. ECS Support of HDF-EOS • Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline. • OS’s: Solaris (9, 10), Irix6.5 (64, n32), Linux (including 64-bit Opteron and Itanium), Mac OS X (Power PC and Mac Intel), Windows XP(MTD Toolkit, HDF-EOS2 & 5, HEG, HDFView) • added full Cygwin support for next release • Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++, Intel Fortran (SGI Altix), gfortran (Mac Intel)

4. Downloadable Software SDP/MTD Toolkit (Science Data Processing Tools) HDF-EOS2 (HDF4 based) HDF-EOS5 (HDF5 based) heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents)

5. Downloadable Applications HEG - GeoTIFF conversion - Spatial subsetting, Field Subsetting - reprojection - stitching, sub-sampling and more …. DumpMetadata (Dump HDF-EOS metadata) HDFView Plug-in, Java-based browser (HDF4, HDF5, HDF-EOS 2 and 5 access) HE5View*(HDF5 - based browser) EOSView*(HDF4 - based browser) *Dropped support for OS updates on July 2006

6. Software Releases Toolkit/HDF-EOS: Last release – March 2008 TOOLKIT 5.2.15 HDF-EOS 2.15, HDF- EOS5.1.11 with HDF4.2r3, HDF5-1.6.7, Szip2.1 Next release – May 2009 TOOLKIT 5.2.16 HDF-EOS 2.16, HDF-EOS5.1.12 with new releases of HDF4 (2r4), 5 (1.8.3)

7. Software Releases (Cont.) HEG (HDF-EOS2 to GeoTIFF Converter): Last release – August 2008 Version 2.9 Next release – May 2009 Version 2.10 Already have released Datapool version to the DAACs (Feb. 2009)

8. Software Releases (Cont.) HDF-EOS Plug-in for HDFView: Last release – September 2008 Version 2.4 Next release – June 2009 For HDFView Version 2.5

9. Software Releases (Cont.) HECONVERT (HDF-EOS2 to HDF-EOS5 Converter): Last releases – 2001, 2004 Version ? – Oct. 2008 Version 2.0 DumpMetadata (ASCII Dump of HDF-EOS metadata) Last release Dec. 2008 Version 1.0

10. SDP Toolkit • Science Data Production (SDP) Toolkit is a collection of tools used by - Science software developers who produce code to process instrument data. - HDF-EOS userswho need to • Access metadata in HDF-EOS files • Perform Time/Date conversion • Access Digital Elevation Model Data • Access Ancillary Data, such as Digital Chart of the World database (DCW), Olson World Ecosystem files, DEM data files, etc.

11. SDP Toolkit (cont.) • Tools provided in (SDP, MTD) Toolkit: - AA (Ancillary Data Access) - CBP (CELESTIAL Body Position) - CSC (Coordinate System Conversion) - CUC (Constant and Unit Conversions) - DEM (Digital Elevation Model Access) - EPH (Ephemeris Data Access) - GCT (Geo Coordinate Transformation) - IO (Input Output (File I/O)) - MEM (Memory Management) - MET (Metadata Access) - PC (Process Control) - SMF (Status Message File (Error/Status)) - TD (Time and Date Conversion)

12. What is HDF-EOS ? • HDF-EOSis an extension to HDF which standardizes storage of and access to common Earth Sciences data structures • By providing a single interface to data structures common to earth science • By providing a container for EOS inventory, archive and product specific metadata • By establishing a consistent relationship between geolocation and science data.

13. HDF-EOS 2 • HDF4 - based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • HDF4 and HDF-EOS 2 will be supported by THG/NASA for the fore-seeable future.

14. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF-EOS2 with a different user interface. • First released in 2000. • Format used byEOS Aurainstruments • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. • Supports same data structures • Added prefix ‘HE5_’ to HDF-EOS 2 functions.

15. HDFView • HDFView: A Java-based browser from THG for HDF4 and HDF5 – based data. • HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions. • The extension is a plug-in module and does not change THG HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains.

16. HDFView • Identical look and feel when displaying HDF and HDF-EOS objects. • Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection info., dimensions, datafields, and attributes are displayed.  • Current Support: SUN, SGI, LINUX, Mac, WINDOWS platforms

17. Swath, Grid, and Point Objects

18. AMSR-E Swath Tableview Features in Version 2.4

19. HDF-EOS to GeoTIFF Converter (HEG) HEG: Utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats. • Two versions available: • A downloadable desktop version • Access through NASA archive online storage (Data Pools) • Implemented also in some prototypes: • Stand-alone Data Pool • OGC (Open Geospatial Consortium) for Web Services Chaining • DOWS (Deploy OGC Web Services) on the DAACs

20. HEG Functionality • Conversion: • HDF-EOS (Swath or Grid ) to a single-band or multi-band GeoTIFF’s • HDF-EOS Swath toHDF-EOS Grid • HDF-EOS Swath or Grid to generic Binary • Metadatapreservation/creation. • Allows file selection from users local storage

21. HEG Functionality • Subsetting (spatial, field, band) • Stitching (mosaicing) + subsetting + Reprojecting • Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN, Albers) • Subsampling (Subsample stacks) • Running from Command Line (batch jobs) or with Java-based GUI

22. HEG Functionality • Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products), • Operable on Sun, SGI, Win, Linux, MAC • Integrated into ECS Data Pool • Reduces the transfer time of HDF-EOS data sets (if subset is requested) • Provides the end-user with the exact file required by their application • User Interface • Portable, written in Java. • Not dependent on COTS (eg. IDL).

23. Mt. Etna Eruption: MISR L1B Stitched Blocks

24. 2004 Hurricane Francis, MODIS MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5

25. AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF, (3 bands: Visible Near Infrared (VNIR)) ….. and Subsetted Two Swaths Stitched …..

26. TOOLKIT/HDF-EOS: SZIP Insatallation: Default decoder, Choice for encoder Choice for configuring HDF4 with/without SZIP Choice for disabling HDF4’s internal netCDF Choice for installing Toolkit/HDF-EOS in 32 or 64-bit modes in 64-bit Linux platforms (except for Itanium; no –m32 flag) Recent Major Development/Fixes

27. Support for new releases of HDF4/HDF5/SZIP (HDF4.2r4, HDF5-1.8.X, SZIP-2.1) Auto Configuration capability (like HDF) for HDF-EOS2 and HDF-EOS5 Support for Mac Intel platform. Support for CYGWIN Recent Major Development/Fixes

28. Added APIs GDij2ll and GDll2ij to HDF-EOS2 and HDF-EOS5 Added HDF_EOS 2 & 5 utilities for dumping grid lat/lon into ASCII or binary files Bug fixes such as: - HDFEOS5: Attribute data type inquiry (code returned not useful for reuse) - HDFEOS5: Tiling/comp. consistency between dataset and structmetadata - HDF-EOS2: Correct diminfo for a field (value in structmetadata is wrong?) - HDF-EOS5: wrong datatype info for INT8 data (C code) - Toolkit: Problem in pccheck utility( hangs if a blank space at a line start ) - Fix for ODL core dump (too many file names for Input Pointer) Recent Major Development/Fixes

29. HDF-EOS Plug-in for HDFView: Current Support: SUN, LINUX, WINDOWS, and MAC (Power and Intel) Auto installation of plug-in into HDFView running self-extracting archives. Adding to plug-in the fixes and enhancement that went into HDFView2.4 and 2.5 Display image for selected pixels in the viewed tables Recent Major Development/Fixes

30. Display Lat/Lon for a selected pixel in the data field tables Full Mac OSX support Improved Help Window New icons for activating Lat/Lon display for Swath & Grid Format conversion: HDF-EOS2 -> HDF-EOS5 Correction for compression and tiling values for grid fields Recent Major Development/Fixes

31. HEG: Product Support: AMSR-E: Land products (CEA Projection added) AMSR-E 5 day Snow, and Monthly Snow Weekly Ocean, and Monthly Ocean Daily 12 Km, and Daily 6 Km Sea Ice L2 Rain, and Rain Grid ASTR_07XT, ASTR_09XT, ASTR_L1BE, MOD29E1D More new versions and products tested. Will update Users’ Guide and HEG web site Recent Major Development/Fixes

32. Stitching fixes: MOD14A1 3-D fields (band selection) TM grids Multi-band geotif/HDF-EOS in Datapool Other Fixes: Interpolation correction with AIRS products containing fillvalue in the data fields Pixel size unit consistency in conversion and stitch GUIs. Recent Major Development/Fixes

33. Correction for droping a column when stitching MODIS SIN tiles. Fixes for processing with default pixel size instead of ignoring it for Geographic projection Spatial subsetting fix for global AMSR-E garanules Fix for ignoring re-projection of type NN interpolation after stitching support pixel resampling for gridded data Recent Major Development/Fixes

34. Future Plans TOOLKIT/HDF-EOS: Auto Configuration for TOOLKIT XML output for Metadata in TOOLKIT Update of DEM land-sea mask Support for HDF5-1.8.X Validation tool for HDF-EOS5 files

35. Future Plans (cont.) HDF-EOS Plug-in for HDFView: More functionalities : - Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes. - Ability to create new grid/swath HEG: - Support for AMSR-E Level 3 Land (CEA projection) - New MRT (MODIS Reprojection Tool) release updates - Support HEG for MAC Intel

36. Availability Access to libraries, applications, Documentation: TOOLKIT: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html HDFView: http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/ HDFView_hdfeos_plugin.html HEG: http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html Email Abe_Taaheri@raytheon.com Landover_PGSTLKIT@raytheon.com