RFCs for HDF5 and HDF-EOS5 Status Update

1. RFCs for HDF5 and HDF-EOS5Status Update Richard Ullman Chair ES-DSWG - Standards November 29, 2006

2. Overview • What is the NASA Earth Science Data System Working Group, Standards Process Group? • HDF in the SPG • Invitation to comment.

3. Motivation • One initiative after another has stressed the need for interoperability standards. • Many standards initiatives, both formal and grass roots have put forward specifications or demonstrated various ways to enable access to data. • NASA, or NASA funded projects are often in the forefront of these activities. • However, NASA participation in a standards development activity does not imply that NASA projects endorse the results of that activity. • Need a way to identify the “standards that work” in the context of NASA’s Earth science research and applications data systems under the operational loads that NASA typically experience.

4. EOSDIS Evolution 2015 Vision Tenets

5. Group on Earth bservations A Shared Vision for Earth ObservationArticulated by 34 Nations in an Earth Observation Summit (July 31, 2003) An international comprehensive, coordinated and sustained Earth observation system • Comprehensive: meeting the needs of a variety of science and applications disciplines • Coordinated: multinational satellite, suborbital and in situ observing capabilities strategically coordinated via agreed standards and data exchange • Sustained: long-term, continued financial and in-kind support from funding authorities

6. Insights from “SEEDS” Analysis • Interoperability does not require homogeneous systems, but rather coordination at the interfaces. • Management can judge success based upon program goals rather than dictated solutions. • example: degree of interoperability rather than use of particular data format. • Communities of practice have solutions. • Published practices that demonstrate benefit can grow … • successful practice in specific community • broader community adoption • community-recognized “standards”

7. The Request For Comment Process • Modeled after example of Internet “IETF RFC”. • Tailored for responsiveness to NASA. • Proposed standards are documented as specifications acording to SPG guidelines and submitted by practitioners within the NASA community. • The Standards Process Group forms a Technical Working Group (TWG) to coordinate evaluation. • What does “implementation” of this specification mean in the context of NASA Earth Science Data Systems? • What constitutes successful “operational” experience? • The community is invited by means of email announcement to comment on the specification and particularly to address questions formulated by the TWG. • The TWG also identifies key stakeholders that are likely to have particular experience with the technology and solicits their opinion. • The TWG reports to the SPG and the SPG makes recommendations for final status of the RFC.

8. The Review Process • Initial Screening • Initial review of the RFC • Provide RFC submission support • Form TWG; set schedule RFC Proposed Standard • Stakeholder/TWG Review • Specification • Operational Readiness • Suitability for Use Stakeholders Respond to TWG Develop review criteria Evaluate known experiences Evaluate community response TWG Recommendation SPG Recommended Standard

9. The Endorsement Process • SPG will send recommended standards to NASA HQ Program Executive for Data Systems with the following: • Strengths/ Weaknesses • Applicability/ Limitations • Endorsement is briefed to HQ Earth Science Steering Committee • HQ will disseminate endorsement through NASA CIOs and to general announcement email to community • “Core” standards will be required of NASA Earth Science programs, projects and awards. • “Community” standards will be applied at discretion of program or studies managers.

10. RFC criteria: • Are there components (technologies practices) that if documented and more widely used would promote: • Easier sharing or exchanging of data among distributed partners and users. • Distributed systems development and sharing of software and technical expertise. • Reducing the cost of developing or maintaining a system. • Increasing the use of scientific data products and bringing more funding. • Interoperability and enhancing innovation, collaboration, and computing performance. • For identified technologies/practices, Is there a community of use that: • Has experience in implementation and operation . • Has leadership necessary to promote the advantage of wider use.

11. HDF5 and HDF-EOS RFCs in the NASA SPG Process • Related RFCs handled by same TWG • One review request for both RFCs • Community members asked to comment on either or both of the RFCs • Target community includes • NASA instrument teams • NASA data systems managers • Earth science investigators • Developers of software tools for Earth science data • Comment period for “implementation experience” April 26 – July 31

12. HDF RFC “Implementation” Reviewers • Total of 17 people commented on one or both RFCs • Backgrounds include: • Data providers (AIRS, HIRDLS and MLS instrument teams) • NASA data systems managers (ORNL and NSIDC DAACs, TSDIS/GPM, OMI SIPS ) • Developers of software tools for Earth science data (RSS – developers of IDL, UAH Subsetting team, Unidata, MathWorks, NetCDF, HDF Group (reviewing HDF-EOS5), HDF-EOS team (reviewing HDF5)) • Earth science data users (CERES instrument team, DSCOVR project) • All using HDF and/or HDF-EOS software from original developers, perhaps with minor modifications • Tools used include FORTRAN, IDL, Mathematica, Scilab, HDFLook, HDFView, Matlab • Operating systems include Windows (32 and 64 bit), Linux (32 and 64), Solaris (32 and 64), Mac (PPC & Intel), SGI, IRIX, HP-UX

13. Summary of comments from Implementation Review phase • HDF5 comments from 10 reviewers were generally positive • 9 of 10 recommending endorsement • HDF-EOS5 comments from 8 reviewers were somewhat mixed, still overall positive • 4 recommending endorsement • 2 neutral • 2 recommending improvements to RFC before endorsement • Primary concerns: • Complexity of data model, API and software libraries • Availability of tools to improve ease of use • Long term support

14. Other Comments Received • Comments on HDF in general from 4 reviewers • General comments, not addressing implementation questions • Not specific to either flavor of HDF being considered • Maybe these should be considered for “operations” rather than “implementation” phase • 3 of 4 of these comments were negative • Both Science Data Users fell in this group

15. Summary of Comments So Far from Usability Survey • Total of 13 people have commented on HDF5 • 12 are software tools developers (among other roles); 1 is a data user only; 7 classify themselves as both • 5 are international • US respondents from military, national labs, high performance computing, NOAA, commercial companies • Should HDF be a recommended format for NASA Earth Science data? • 10 selected “strongly agree” • 1 selected “disagree” • 1 selected “neutral” • 1 selected “not applicable”