Prepared By: Tom King 1 , Zhaohui Cheng 1 , Larisa Koval 1 , and Yi Song 2 , 1 PSGS, 2 IMSG

1. NetCDF4 Reformatting Toolkit (N4RT): BUFR and GRIB2 TailoringCritical Design ReviewSeptember 14, 2009 Prepared By: Tom King1, Zhaohui Cheng1, Larisa Koval1, and Yi Song2, 1 PSGS, 2 IMSG

2. Review Agenda Introduction 9:00 am – 9:20 am Cheng PDR Report 9:20 am – 9:40 am Cheng Requirements 9:40 am – 10:10 am King Software Architecture 10:10 am – 10:45 am King Quality Assurance 10:45 am – 11:05 am King Risks and Actions 11:05 am – 11:20 am Cheng Summary and Conclusions 11:20 am – 11:30 am King

3. Review Outline • Introduction • PDR Report • Requirements • Software Architecture • Quality Assurance • Risks and Actions • Summary and Conclusions

4. Introduction Presented byZhaohui Cheng NOAA/NESDIS/STAR

5. Introduction • Project Background • IJPS • NPP/NPOESS • NDE • Project Objectives • Integrated Product Team • Project Plan • Entry and Exit Criteria

6. Project BackgroundNPP/NPOESS • NPP and NPOESS, a joint Military/NOAA/NASA effort, is the next series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the NPOESS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (1:30 pm). • Instrument packages on NPOESS: • CrIS, ATMS, VIIRS,OMPS, SEM, CERES, MIS • NPP is the first of five missions with launch dates of ≈2011, ≈2013, ≈2016, ≈2018, ≈ 2020, respectively.

7. Project BackgroundNDE • Disseminate NPOESS Data Records to customers. • Generate and disseminate tailored NPOESS Data Records (versions of NPOESS Data Records in previously agreed alternative formats and views). • Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPOESS Data Records). • Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving. • Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA. • Provide software for NPOESS Data Record format translation and other data manipulations.

8. Project Objectives • To build a software package that will tailor NPOESS and NDE products from NetCDF4 into BUFR and GRIB2 formats in support of NDE’s overall tailoring efforts. • The NetCDF4 Reformatting Toolkit (N4RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products. • Flexible • Extendable • The software must be able run within the NDE system architecture and operate within the NDE functional guidelines. • Output product formats and content must meet the needs of NOAA customers.

9. Project Objectives Phase 1 Products • Products to Reformat • ATMS Radiances • CrIS Radiances • Nadir Profile and Total Column Ozone (OMPS) • VIIRS Radiances • Aerosol Optical Thickness • Sea Surface Temperature • Snow Cover (Currently Tabled) • Vegetation Index (Currently Tabled) • Polar Winds (Possibly New) • Green Vegetation Fraction (Possibly New)

10. Integrated Product Team • IPT Lead: Walter Wolf (STAR) • IPT Backup Lead: AK Sharma (OSDPD) • NESDIS team: • STAR: Walter Wolf, Hank Drahos, Jaime Daniels, Yi Song, Thomas King, Larisa Koval • OSDPD: Dave Benner, AK Sharma, Ricky Irving • OSD: Tom Schott, Jim Yoe • Data Center: Lei Shi (NCDC) • User team • Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber (NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard (JCSDA), Tony McNally (ECMWF), Fiona Hilton (UK-Met) • Others: International NWP users, NWP FOs, Climate Users • Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP

11. Project Stakeholders • NOAA National Weather Service • Weather Forecast Offices • National Center for Environmental Prediction • Department of Defense • NRL • FNMOC • AFWA • Global NWP • EUMETSAT • ECMWF • UK Met • Meteo France • CMC

12. Project Plan • Year 1 – Design and Development (2008 – 2009) • Verify Requirements • Work with customers to verify product requirements • Discuss with the current developers of similar translators to determine what is required in their output files • Design the NetCDF4 reformatting toolkit; • Conduct PDR • Develop BUFR tables and GRIB2 formats with the product teams for Phase 1 products • Work with NDE to determine the interface between the Level 1B and the Level 2 NPP products and the reformatter • Conduct CDR

13. NetCDF4 Reformatter: System Design and Development for Phase 1

14. Project Plan • Year 2 –Transition to Pre-Operations of Phase 1 Products (2009 – 2010) • Set up infrastructure to implement the readers and writers for the data formats • Implement BUFR tables and GRIB2 formats for the Phase 1 products on the NDE hardware • Conduct Test Readiness Review for Phase 1 products • Transition and test system within the NDE environment • Conduct Code Review for Phase 1 products

15. Project Plan • Year 3 – Transition to Operations of Phase 1 Products (2010 – 2011) • Evaluate with NDE and OSDPD the implementation of the Reformatting Toolkit within the NDE data handling system • Conduct System Readiness Review for Phase 1 products • Transition pre-operational Phase 1 product reformatting system to operations

16. Phase 1 Transition to Operations

17. Project Plan – Schedule • Schedule (Milestones) • Project begins – 07/01/08 • Preliminary Design Review – 04/14/09 (10/21/08) • Critical Design Review – 09/14/09 (03/19/09) • Test Readiness Review – 04/14/10 (02/25/09) • Code Unit Test Review – 05/12/10 (01/29/10) • System Readiness Review – 01/31/11 (04/20/10) • Waive or shift to NDE

18. Entry Criteria • PDR Completed and the following items reviewed: • Requirements • Software Architecture • Quality Assurance • Risks and Actions • Requirements Allocation Document - Updated • PDR Report • The PDR Report (PDRR) is a standard artifact of the STAR EPL process. The PDR report is produced after the PDR. It contains: • Actions • Comments • Revision of the PDR • PDR Report for this project contains 13 items for review. These will be discussed in the next section.

19. Exit Criteria • Conduct the CDR • The CDR Report (CDRR) is a standard artifact of the STAR EPL process. The CDR report is produced after the CDR. It contains: • Actions • Comments • Revision of the CDR

20. Summary of Review Objectives • Review the Project Schedule and Plans • Review of the PDR Report • Review the Requirements • Review Software Architecture • Review Quality Assurance • Identify Risks and Actions

21. Review Outline • Introduction • PDR Report • Requirements • Software Architecture • Quality Assurance • Risks and Actions • Summary and Conclusions

22. PDR Report Presented byZhaohui Cheng NOAA/NESDIS/STAR

23. PDR Risk • Risk 1: NPOESS product formats and content are still changing, especially for VIIRS • Assessment: Low • Impact: May have to revise software several time during development to adjust to new formats, names, and types. • Likelihood: High • Mitigation: Work through the NPOESS Data Format Working Group to obtain information on format and algorithm updates. Monitor the latest copies of the NPOESS Common Data Format Control Books (CDFCB) for any updates. Maintain contact with customers to inform them of any upstream product changes. Make the code design flexible so that changes in the upstream products translate into the minimum amount code revision. • Status: Open

24. PDR Risk • Risk 2: The roles and responsibilities regarding who shall generate the set of required SPSRB documents for NDE has not yet been decided. • Assessment: Low • Impact: Difficult to budget time needed for the team to generate documentation. • Likelihood: Moderate • Mitigation: This issue, and that of document content, is being worked by Maurice McHugh, STAR, NDE, OSD, and OSDPD personnel. Reformatting Toolkit developers intend to participate in these meetings and discussions. • Status: Open

25. PDR Risk • Risk 3: There are small variations in the types of platforms and the versions of the compilers • Assessment: Low • Impact: May obtain different results using different compilers. • Likelihood: Moderate • Mitigation: Reformatting Toolkit developers will work with NDE during system tests in the integration and production phase to ensure that those results match the results from the units. The BCR process (mentioned earlier) should help to resolve these issues early on in development. • Status: Open

26. PDR Risk • Risk 4: Data format translation involves some unit conversion and possible reduction of precision (significant digits) • Assessment: Low • Impact: Any modification of the data from its original form may not be apparent to the user. • Likelihood: Low • Mitigation: Document data manipulations in the NDE Delivered Algorithm Package (in place of ATBD). • Status: Open

27. PDR Risk • Risk 6: Risk on NDVI and snow mask. Test GRIB2 functionality before/for CDR. • Assessment: Moderate • Impact: Failure to meet the requirement to have demonstrated GRIB2 tailoring capability. • Likelihood: Low • Mitigation: RAD/Requirements meeting before CDR. • Status: Open

28. PDR Risk • Risk 8: Requirement to write tailoring into GRIB2 VIIRS snow cover and vegetation index. Need to insure that view point M. EK is known and understood by EMC director. • Assessment: Moderate • Impact: User (EMC) does not receive the product they requested. • Likelihood: Moderate • Mitigation: Yoe/Schott to contact M. EK and coordinate w/s Lord/EMC/JCSDA before decisions (i.e. snow cover and vegetation index may decide differently) • Status: Open • Comments: Has this been done? If so, it can be closed.

29. PDR Risk • Risk 9: NWS and other NWP users will want ATMS/CrIS spectral response functions. They will also want OMPS key calibration functions. This information appears to be contained In ITAR document that we cannot share with foreign users. • Assessment: Moderate • Impact: Users will not be able to use the data. • Likelihood: High • Mitigation: NDE should work with IPO to determine if we can receive and release that ATMS, CrIS and OMPS SPF and KCF data to foreign NWP users. It is our understanding that the information is in ITAR documentation • Status: Open • Comments: We recommend closing this risk since it is not a risk to the Tailoring project. It is a user need that would be best addressed by IPO.

30. PDR Risk • Risk 11: A lot of effort would be imposed on projects/NDE to convert to NetCDF4 (depend heavily on NDE schedule for the conversion to NetCDF4). • Assessment: Low • Impact: Increased schedule pressure on NDE Product Teams. • Likelihood: Low • Mitigation: Make converter tool flexible to accept other formats not just NetCDF4. Those formats should include:1) project formats (MIRS, NUCAPS, Ozone etc.) 2) HDF5 format from IDPS. This is not a risk for the Tailoring project. This is an NDE or Product Team risk. • Status: Open • Comments: We recommend closing this since it is not a risk to the Tailoring project. It’s a risk to NDE and the Product Teams.

31. PDR Risk • Risk 12: Timelines gets tight If we have to convert HDF5, project format to NetCDF4 and then to other format. • Assessment: Moderate • Impact: Users may receive their product later than desired. • Likelihood: NA • Mitigation: Asses timeliness issue (due to added conversion step). • Status: Open • Comments: We recommend closing this since it is not a risk to the Tailoring project. It’s a risk to NDE and the Product Teams.

32. PDR Comment • Review Item 5: Requirement section: traceability to L1 requirements is a strength. This section mingles project requirements, system requirements and assumptions. • Mitigation: In the RAD and PDR, clearly identify assumptions up-front, including expectations of the NDE system. Remove "NDE shall…" requirements. NDE team will validate assumption and assume new requirements if necessary. • Comments: The PDR and RAD have been revised to identify only the Tailoring project’s requirements. The language in the requirements analysis has been modified to express the project’s understanding of NDE’s roles.

33. PDR Comment • Review Item 7: Set up a meeting with Tom Schott on new aerosol product requirements/requests. • Comments: Has this been done?

34. PDRR – Item 10 • Review Item 10: Minor terminology and details. 1) IPs: Not all IPs are created equal. DIPs: Delivered IPs (OMPS NP O3 Profile). RIPS: Retained IPs 2) O3 POP is now Atmospheric Chemistry POP. 3) OMPS Total Ozone Algorithm is multiple triplet Not v8 primary. Output fields are equivalent to v8 on GOME-2. • Mitigation: 1) Not existence of DIPs. Current OMPS DIP is v6. 2) Change in POP list. 3) Get sample EDR compare fields to GOME-2 v8 PMF • Comments: The PDR has been updated to contain the enhanced IP definitions and ozone algorithm version.

35. PDR Comment • Review Item 13: Good idea to have placeholder for additional quality flags (for land mask etc.). Filling them depends on timelines. Why not the same thing is done to ATMS BUFR table. • Mitigation: Plan for placeholders of additional quality flags for ATMS radiance BUFR table. The need maybe there but not expressed. • Comments: There are already spare bit fields in the ATMS BUFR quality flag fields if new ATMS quality flags become available. With respect to the tailoring software, it’s not within the scope of this project to assess data quality such that we generate and add to the BUFR our own new quality information.

36. PDRR Summary • There are 9 risks and 4 comments at this time. • 5 risks are rated as low, 4 as moderate. • We recommend closing 3 of the risks. • 6 risks will remain open. • An additional risk may be closed depending upon the status of progress (meetings Yoe/Schott and EMC).

37. Review Outline • Introduction • PDR Report • Requirements • Software Architecture • Quality Assurance • Risks and Actions • Summary and Conclusions

38. Requirements Presented byThomas King NOAA/NESDIS/STAR

39. Requirements Overview • SPSRB Requirements were presented to the developers in a document entitled: “Level 1 Requirements for a NetCDF4 Reformatting Tool” (Version 1.5). • Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, EMC (the customer) and the heritage product teams. • Using all of this information a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project. • Text highlighted in yellow indicates requirements that have been updated or refined since the PDR.

40. Phase I User-to-STAR Mapping

41. Functional Requirements:Reformatting Toolkit Software • Requirement: STAR shall deliver to NDE a reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2). • Requirement: The toolkit shall be capable of reformatting the NPP tailoring prioritized phase 1 product list.

42. Functional Requirements:Reformatting Toolkit Software • Requirement: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment. • The Toolkit shall compile and run on the NDE IBM AIX P5 series hardware. • The Toolkit shall interact with the NDE Data Handling System (DHS). • The Toolkit shall be able to read a Production Control File (PCF). • The Toolkit shall handle and return errors according to NDE/STAR standard codes. • The Toolkit shall be able to write a (Product Status File) PSF. • Requirement: The toolkit shall consist of modular components that can be tested independently. • The code shall consist of a single compiled program that parses arguments and logically assigns tasks to a family of hierarchically structured tailoring subroutines. • Data shall be stored in allocatable data structures.

43. Functional Requirements:Reformatting Toolkit Software • Requirement: STAR shall include one update to the reformatting toolkit within its initial project plan. • Requirement: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products. • Requirement: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit. • Requirement: STAR shall update the reformatting toolkit when NCEP updates its BUFR and GRIB2 libraries. • Updates shall be made when there are updates to the versions of the NetCDF4 library being used by NDE.

44. Functional Requirements:Reformatting Toolkit Software • Requirement: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development. • Requirement: The output from the toolkit shall be compared with the input. • Requirement: The translation toolkit shall convert from the new format back into NetCDF4. • Requirement: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes. • The Reformatting Toolkit software shall generate run logs and return NDE/STAR standard (agreed upon) error codes to the DHS.

45. Functional Requirements:Reformatting Toolkit Software • Requirement: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination). • Requirement: The toolkit parameters (e.g., how to use the service) shall be well documented. • Reformatting Toolkit Developers shall provide documentation in the form of a tailored Delivered Algorithm Package (DAP). • Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators. • Reformatting Toolkit shall use the standard set of error return codes developed by NDE for code running within the DHS.

46. Functional Requirements:Reformatting Toolkit Software • Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting. • All messaging shall be directed to a run log file. These messages shall be documented in the Reformatting Toolkit tailored DAP. • Requirement: STAR shall coordinate development of the reformatting toolkit with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system. • Requirement: Toolkit code shall adhere to the STAR coding standards. • Requirement: Performance shall be measured on a product level.

47. Program Requirements:Reformatting Toolkit Project • Requirement: STAR shall provide monthly project status reports to OSDPD and OSD. • Requirement: Earned Value Management shall be performed on the project. • Requirement: STAR shall update the project plan on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration. • Requirement: The toolkit shall be implemented and tested six months before the NPP launch to ensure NDE readiness.

48. Product Requirements:CrIS Radiances • Requirement: The Reformatting Toolkit shall tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC. • The Reformatting Toolkit developers shall work with EMC to create a BUFR table for the NUCAPS thinned radiances based on AIRS and IASI. • The table shall use delayed replication for storing the radiances. • BUFR messages shall be smaller than 50KB. • The BUFR format shall allow for the storage of negative radiances. • The file shall contain the following data fields (see table next slide):

49. Product Requirements:CrIS Radiances

50. Product Requirements:ATMS Radiances • Requirement: The Reformatting Toolkit shall tailor the NPOESS ATMS SDR and TDR data from NetCDF4 into BUFR for EMC. • The ATMS BUFR file shall contain, from all channels, the antenna and brightness temperatures, associated Quality Flags, and the geolocation data at native resolution (not resampled) data. • The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is currently provided for AMSU and MHS. • BUFR messages shall be smaller than 50KB. • The file shall contain the following data fields (see table next slide):