ESDIS 2008 tasks status

1. ESDIS 2008 tasks status Kent Yang, Mike Folk The HDF Group March 31, 2009 Annual briefing to ESDIS

2. Summary of tasks • Total 28 tasks: • Most tasks received from the annual briefing and the workshop • 4 NPP/NPOESS tasks transferred to NPOESS project • 10 tasks have been finished • 6 tasks are on-going activities • 8 tasks are in progress Annual briefing to ESDIS

3. Finished tasks highlights Annual briefing to ESDIS

4. HDF-EOS automatic configuration • Both HDF-EOS2 and HDF-EOS5 • Added the szip support • Both encoder and decoder • Decoder only • Without szip • Added Fortran wrapper support • Updated source code under Fortran testdrivers directory • Updated installation files for automatic configuration Annual briefing to ESDIS

5. Investigation of HDF Support in Some Open Source Software Packages Annual briefing to ESDIS

6. Five open source packages • PyHDF • Python interface to HDF4 • http://pysclint.sourceforge.net/pyhdf/ • Geospatial Data Abstraction Library (GDAL) • Translator library for Raster Geospatial Data Formats • Supports about 100 file formats • http://gdal.org/ • NCAR Common Language (NCL) • Interpreted Language for Data Analysis and Visualization • http://ncl.ucar.edu/ • Grid Analysis and Display System (GrADS) • Interpreted Language for Data Analysis and Visualization • http://iges.org/grads/ • GNU Data Language (GDL) • Interpreted Language for Data Analysis and Visualization • Data Analysis and Visualization • http://gnudatalanguage.sourceforge.net/ Annual briefing to ESDIS

7. Evaluation - File Formats 9/13/2014 Annual briefing to ESDIS 7

8. Evaluation - Documentation 9/13/2014 Annual briefing to ESDIS 8

9. Evaluation - Installation 9/13/2014 Annual briefing to ESDIS 9

10. Usage example - GrADS $ grads ga> xdfopen AE_RnGd.xdf ga> set lon -180 180 ga> set gxout shaded ga> display RrLandRain ga> draw title Total Rain Rate over Land in July 2007 Annual briefing to ESDIS

11. Performance evaluation of using SSHFS-FUSE to access HDF5 files Annual briefing to ESDIS

12. Using NFS remote system local system • NFS requires a special permission. General users are not allowed. NFS Annual briefing to ESDIS

13. 1. Downloading a whole file remote system local system ftp, sftp, … • What if only a small part is necessary from a huge file? local filesystem Annual briefing to ESDIS

14. 2. Using FUSE+SSHFS remote system local system • Given that permission to access FUSE is granted, general users can mount remote filesystems. SSHFS FUSE Annual briefing to ESDIS

15. Accessing HDF5 through SSHFS-FUSE • Experiment: • For 4 files, • Copy a whole file • Copy one dataset in a file • Copy one small hyperslab in a dataset Annual briefing to ESDIS

16. Results of the experiment • elapsed time ratio • - SSHFS / downloading a whole file and subsetting SSHFS consumed more time SSHFS consumed less time Annual briefing to ESDIS

17. Other finished tasks • The third dimension support of HDF-EOS • The documentation of HDF-EOS needed to be updated • HDF4.2r4 release • Static library with static external libraries • C shared libraries • HDFView 2.5 release • Fixed bugs of ‘switch base index’ and ‘fill value’ • Help build EOS toolkit on solaris • NetCDF4/NetCDF3/HDF5 performance study • http://www.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf Annual briefing to ESDIS

18. On-going tasks overview Annual briefing to ESDIS

19. On-going tasks overview • Applications/other technologies(in separate talks) • Support OPeNDAP • HDF4 handler for OPeNDAP • Independent HDF4 mapping • HDF5 library • Support backward/forward compatibility • Tests with other software packages • Maintain HDF-EOS2 and HDF-EOS5 daily tests • Regression tests with IDL and Matlab Annual briefing to ESDIS

20. Tasks in progress • Some tasks almost finished • Some tasks we need to know the next step • Some tasks are re-scoped into our next year’s plan Annual briefing to ESDIS 9/13/2014 Annual briefing to ESDIS 20

21. Special value RFC • 90% done. • We have written the RFC • We then sent out to NASA users and hdf-forum • A decision has been made based on feedback • Provide examples to demonstrate how to handle special values Hi Kent, Thank you for your message. I appreciate your taking the time to communicate with us. I appreciate even more that your decision was guided by the needs, concerns, and interests expressed by hdf5 data producers. - Paul Annual briefing to ESDIS

22. High Level Language Support • Motivation • A better high-level language support for HDF5 such as Python, Perl, Ruby etc. • Progress • Investigated SWIG-HDF5 interface • Held a discussion in HDF-EOS workshop • Formed a High level language wiki to further discuss the general issue http://wiki.hdfgroup.org/High+level+language+supports+for+HDF5 • Suggestions ? Annual briefing to ESDIS

23. Simplifying installation procedures • Request • Investigate simplifying build installation procedures, including 3rd party libraries • Progress • The HDF group is investigating the use of RPM to simplify the whole process • A RFC is being written Annual briefing to ESDIS

24. Using NetCDF4 APIs to read HDF-EOS5 • Motivation: • To allow the access of HDF-EOS5 files via NetCDF4 APIs • To ensure that HDF-EOS5 files still accessed by the HDF-EOS5 library Annual briefing to ESDIS

25. Using NetCDF4 APIs to read HDF-EOS5 • Progress: • Wrote an investigation report on this issue • Discussed with Aura, netcdf and EOS developers • Finalized a solution • Augmented the EOS5 file to support dimensions • Relaxed creation order requirements of NetCDF4 to access HDF-EOS5 data • A prototyped augmentation tool is implemented • Work to be done: • Wait for NetCDF4 developer to release a snapshot to support the relax of creation order • Write a netCDF4 fortran example to read EOS5 files Annual briefing to ESDIS

26. HDF5 to jpeg tool prototype • Request from HDF-EOS5 to support the conversion in HDF-EOS2 to HDF-EOS5 tool • h52jpeg traverses an HDF5 file and saves each HDF5 image found to a jpeg file • h52jpeg is only a prototype • Not fully tested • Not fully documented • Not released with HDF5 library • Source code is available only at a special branch http://svn.hdfgroup.uiuc.edu/hdf5/branches/jpeg_converter/hl/tools/h52jpeg/ • Is there any request to make this tool fully supported? Annual briefing to ESDIS

27. Command line format conversion tools • Request: • In order of priority - netCDF, geotiff, grib and ascii • Progress: • This is a big task. • We sent out a newsletter to ask if there are any available tools; no useful responses received. • We were implementing an HDF-EOS5 to netCDF4 conversion tool. • Questions: • Is this task still necessary? • If yes, what’s the priority of this task? Annual briefing to ESDIS

28. EOS access example codes and tools • Requests: • Tools such as HEG, IDL , Matlab etc. • Well-commented example codes • Progress: • Some experimental work has been done • Plan: • The work is re-scoped to be the focus of the new year Annual briefing to ESDIS

29. Review dimension scale model • Requests: • Review dimension scale model and identify use cases • Plan: • We will start investigating this after the briefing • Any suggestions? Annual briefing to ESDIS

30. Thank You! Annual briefing to ESDIS

31. Acknowledgements This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. Annual briefing to ESDIS

32. Questions/comments? Annual briefing to ESDIS