.NetCDF4 Reformatting Toolkit: BUFR and GRIB Tailoring

1. NetCDF4 Reformatting Toolkit: BUFR and GRIB TailoringPreliminary Design ReviewApril 14, 2008 Prepared By: Tom King1, Walter Wolf3, Lihang Zhou1, Yi Song2, Larisa Koval1, 1 PSGS, 2 IMSG, 3 NOAA/NESDIS/STAR

2. Review Agenda Introduction 9:00 am – 9:20 am Wolf Requirements 9:20 am – 10:00 am King Software Architecture 10:00 am – 10:45 am King Quality Assurance 10:45 am – 11:05 am King Risks and Actions 11:05 am – 11:20 am Wolf Summary and Conclusions 11:20 am – 11:30 am Wolf

3. Review Outline • Introduction • Requirements • Software Architecture • Quality Assurance • Risks and Actions • Summary and Conclusions

4. Introduction Presented byWalter Wolf NOAA/NESDIS/STAR

5. Introduction • Project Background • Terminology • IJPS • NPP/NPOESS • NDE • Project Objectives • Integrated Product Team • Project Plan • Entry and Exit Criteria

6. Project BackgroundTerminology • Raw Data Records (RDRs) • Sensor Data Records (SDRs) • Temperature Data Records (TDRs) • Environmental Data Records (EDRs) • Intermediate Products (IPs)

7. Project BackgroundTerminology • Raw Data Records (RDRs) • Full resolution, digital sensor data, time-referenced and locatable in earth coordinates with absolute radiometric and geometric calibration coefficients appended, but not applied, to the data.

8. Project BackgroundTerminology • Sensor Data Records (SDRs) • Data record produced when an algorithm is used to convert RDRs to geolocated, calibrated detected fluxes with associated ephemeris data. Calibration, ephemeris, and any other ancillary data necessary to convert the sensor units back to sensor raw data (counts) are included.

9. Project BackgroundTerminology • Temperature Data Records (TDRs) • Calibrated antenna temperature records from a microwave instrument such as ATMS. • Environmental Data Records (EDRs) • Data records produced when an algorithm is used to convert SDRs to geophysical parameters (including ancillary parameters, e.g., cloud clear radiation, etc.).

10. Project BackgroundTerminology • Intermediate Products (IPs) • The NPOESS products that are produced as an intermediate step between the SDR and EDR. These are available from the IDPS upon request.

11. Project BackgroundNPP/NPOESS • NPP and NPOESS, a joint Military/NOAA/NASA effort, is the next series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the NPOESS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (1:30 pm). • Instrument packages on NPOESS: • CrIS, ATMS, VIIRS,OMPS, SEM, CERES • NPP is the first of five missions with launch dates of ≈2011, ≈2013, ≈2016, ≈2018, ≈ 2020, respectively.

12. Project BackgroundNDE • Disseminate NPOESS Data Records to customers. • Generate and disseminate tailored NPOESS Data Records (versions of NPOESS Data Records in previously agreed alternative formats and views). • Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPOESS Data Records). • Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving. • Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA. • Provide software for NPOESS Data Record format translation and other data manipulations.

13. Project Objectives • To build a software package that will tailor NPOESS and NDE products from netCDF4 into BUFR and GRIB2 formats in support of NDE’s overall tailoring efforts. • The NetCDF4 Reformatting Toolkit (N4RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products. • Flexible • Extendable • The software must be able run within the NDE system architecture and operate within the NDE functional guidelines. • Output product formats and content must meet the needs of NOAA customers.

14. Project Objectives Phase 1 Products • Products to Reformat • ATMS Radiances • CrIS Radiances • Nadir Profile Ozone (OMPS) and OMPS Radiances • VIIRS Radiances • Snow Cover • Vegetation Index • Aerosol Optical Thickness • Sea Surface Temperature

15. Integrated Product Team • IPT Lead: Walter Wolf (STAR) • IPT Backup Lead: AK Sharma (OSDPD) • NESDIS team: • STAR: Walter Wolf, Hank Drahos, Jaime Daniels, Yi Song, Thomas King, Larisa Koval • OSDPD: Dave Benner, AK Sharma, Ricky Irving • OSD: Tom Schott, Jim Yoe • Data Center: Lei Shi (NCDC) • User team • Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber (NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard (JCSDA), Tony McNally (ECMWF), Fiona Hilton (UK-Met) • Others: International NWP users, NWP FOs, Climate Users • Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP

16. Project Stakeholders • NOAA National Weather Service • Weather Forecast Offices • National Center for Environmental Prediction • Department of Defense • NRL • FNMOC • AFWA • Global NWP • EUMETSAT • UK Met • Meteo France • CMC

17. Project Plan • Year 1 – Design and Development (2008 – 2009) • Verify Requirements • Work with customers to verify product requirements • Discuss with the current developers of similar translators to determine what is required in their output files • Design the NetCDF4 reformatting toolkit; • Conduct PDR • Develop BUFR tables and GRIB formats with the product teams for Phase 1 products • Work with NDE to determine the interface between the Level 1B and the Level 2 NPP products and the reformatter • Conduct CDR

18. Project Schedule – Phase 1

19. Project Plan • Year 2 –Transition to Pre-Operations of Phase 1 Products (2009 – 2010) • Set up infrastructure to implement the readers and writers for the data formats • Implement BUFR tables and GRIB formats for the Phase 1 products on the NDE hardware • Conduct Test Readiness Review for Phase 1 products • Transition and test system within the NDE environment • Conduct Code Review for Phase 1 products

20. Project Plan • Year 3 – Transition to Operations of Phase 1 Products (2010 – 2011) • Evaluate with NDE and OSDPD the implementation of the Reformatting Toolkit within the NDE data handling system • Conduct System Readiness Review for Phase 1 products • Transition pre-operational Phase 1 product reformatting system to operations

21. Phase 1 Transition to Operations

22. Project Plan – Schedule • Schedule (Milestones) • Project begins – 07/01/08 • Preliminary Design Review – 04/14/09 (10/21/08) • Critical Design Review – 09/25/09 (03/19/09) • Test Readiness Review – 06/09/09 (02/25/09) • Code Unit Test Review – 09/10/10 (01/29/10) • System Readiness Review – 01/31/11 (04/20/10) • Waive or shift to NDE

23. PDR Report • The PDR Report (PDRR) is a standard artifact of the STAR EPL process. • The PDR report will be produced after the PDR. • The report will be a critical artifact for the Critical Design Review. • Guidelines for the PDRR are found in STAR EPL process assets

24. PDR Entry Criteria • Requirements Document • Review of NetCDF4 Reformatting Toolkit: BUFR and GRIB • Requirements • Software Architecture • Quality Assurance • Risks and Actions

25. PDR Exit Criteria • Preliminary Design Review Report • The PDR Report (PDRR), a standard artifact of the STAR Enterprise Process Lifecycle (EPL), will be compiled before the CDR • The report will contain: • Actions • Comments • PDR presentation

26. Review Objectives • Review the Requirements • Review Software Architecture • Review Quality Assurance • Identify risks and actions

27. Review Outline • Introduction • Requirements • Software Architecture • Quality Assurance • Risks and Actions • Summary and Conclusions

28. Requirements Presented byThomas King NOAA/NESDIS/STAR

29. Requirements Overview • SPSRB Requirements were presented to the developers in a document entitled: “Level 1 Requirements for a NetCDF4 Reformatting Tool” (Version 1.5). • Text in yellow are those requirements derived from the original SPSRB requirements. • Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, EMC (the customer) and the heritage product teams. • Using all of this information a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project.

30. Original Phase I Product List

31. Original Phase I User/Heritage Mapping

32. Functional Requirements:Reformatting Toolkit Software • Requirement: STAR shall deliver to NDE a reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2). • Requirement: The toolkit shall be capable of reformatting the NPP tailoring prioritized phase 1 product list.

33. Functional Requirements:Reformatting Toolkit Software • Requirement: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment. • The Toolkit shall compile and run on the NDE IBM AIX P5 series hardware. • The Toolkit shall interact with the NDE Data Handling System (DHS). • The Toolkit shall be able to read a Production Control File (PCF). • The Toolkit shall handle and return errors according to NDE/STAR standard codes. • The Toolkit shall be able to write a PSF. • Requirement: The toolkit shall consist of modular components that can be tested independently. • The code shall consist of a single compiled program that parses arguments and logically assigns tasks to a family hierarchically structured tailoring subroutines. • Data shall be stored in allocatable data structures.

34. Functional Requirements:Reformatting Toolkit Software • Requirement: STAR shall include one update to the reformatting toolkit within its initial project plan. • Requirement: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products. • Requirement: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit. • Requirement: STAR shall update the reformatting toolkit when NCEP updates its BUFR and GRIB2 libraries • Updates shall be made when there are updates to the versions of the netCDF4 library being used by NDE. • Requirement: NDE and OSDPD shall implement the updated reformatting toolkit into operations once STAR has updated it with BUFR and GRIB library updates from NCEP.

35. Functional Requirements:Reformatting Toolkit Software • Requirement: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development. • Requirement: The output from the toolkit shall be compared with the input. • Requirement: The translation toolkit shall convert from the new format back into NetCDF4. • Requirement: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes. • The Reformatting Toolkit software shall generate run logs and return NDE/STAR standard (agreed upon) error codes to the DHS. • Run times shall be monitored by the NDE DHS.

36. Functional Requirements:Reformatting Toolkit Software • Requirement: Applications running under either Linux or AIX Operating Systems shallbe able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination). • Requirement: The toolkit parameters (e.g., how to use the service) shall be well documented. • Reformatting Toolkit Developers shall provide documentation in the form of a tailored Delivered Algorithm Package (DAP). • Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators. • Reformatting Toolkit shall use the standard set of error return codes developed by NDE for code running within the DHS. The NDE system shall direct this information to operators.

37. Functional Requirements:Reformatting Toolkit Software • Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting. • All messaging shall be directed to a run log file. These messages shall be documented in the Reformatting Toolkit tailored DAP. • Requirement: STAR shall coordinate development of the reformatting toolkit Application Program Interface with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system. • The Reformatting Toolkit shall work with the NDE integration team to integrate the Reformatting Toolkit software into the NDE DHS. • Requirement: Toolkit code shall adhere to the STAR coding standards. • Requirement: Performance shall be measured on a product level.

38. Program Requirements:Reformatting Toolkit Project • Requirement: STAR shall provide monthly project status reports to OSDPD and OSD. • Requirement: Earned Value Management shall be performed on the project. • Requirement: STAR shall update the project plan on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration. • Requirement: The toolkit shall be implemented and tested six months before the NPP launch to ensure NDE readiness.

39. Product Requirements:CrIS Radiances • Requirement: The Reformatting Toolkit shall tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC. • The Reformatting Toolkit developers shall work with EMC to create a BUFR table for the NUCAPS thinned radiances based on AIRS and IASI. • The table shall use delayed replication for storing the radiances. • BUFR messages shall be smaller than 50KB. • The BUFR format shall allow for the storage of negative radiances. • The file shall contain the following data fields (see table next slide):

40. Product Requirements:CrIS Radiances

41. Product Requirements:ATMS Radiances • Requirement: The Reformatting Toolkit shall tailor the NPOESS ATMS Radiances from NetCDF4 into BUFR for EMC. • The ATMS BUFR file shall contain the TDR (Antenna Temperatures), the associated Quality Flags, and the Geolocation data at native resolution (not resampled) data. • The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is current provided for AMSU and MHS. • BUFR messages shall be smaller than 50KB. • The file shall contain the following data fields (see table next slide):

42. Product Requirements:ATMS Radiances

43. Product Requirements:OMPS Ozone • Requirement: The Reformatting Toolkit shall tailor NPOESS OMPS Ozone products from NetCDF4 into BUFR for EMC. • The product shall contain OMPS Nadir Profile and Total Column (this would be the version 8 ozone algorithm for both products). • The Reformatting Toolkit developers shall work with EMC to develop an OMPS BUFR table based on that currently used for GOME and SBUV. • BUFR messages shall be smaller than 50KB. Note: The project originally planned for OMPS radiances to be included. John Derber (EMC) says that they are not needed.

44. Product Requirements:VIIRS SST • Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS SST products from NetCDF4 into BUFR for EMC. • Product shall contain Skin SST, Bulk SST, Quality Flags, Cloud Mask, and geolocation data. • Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS SST product. • The VIIRS SST BUFR table shall be derived from that currently being used for the AVHRR derived SST (from ACSPO - Advanced Clear-Sky Processor for Oceans). • BUFR messages shall be smaller than 50KB.

45. Product Requirements:VIIRS Radiances • Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Radiances from netCDF4 into BUFR for EMC. • The product shall contain VIIRS radiances for 22 channels with associated quality flags and geolocation. Coverage shall be global. • The product shall contain the land and cloud mask if it doesn’t take too long for the IDPS to generate those EDRs. • Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS radiance product. This table shall be derived from that currently being used for the GAC AVHRR. • BUFR messages shall be smaller than 50KB.

46. Product Requirements:Aerosol Optical Thickness • Requirement: The Reformatting Toolkit shall tailor NPOESS Aerosol Optical Thickness (AOT) from netCDF4 into BUFR for EMC. • The product shall contain the AOT, wavelength of AOT, and Aerosol Size. • Reformatting Toolkit developers shall work with EMC to develop the AOT BUFR table based on what has already been done for MODIS. • BUFR messages shall be smaller than 50KB.

47. Product Requirements:VIIRS Snow Cover • Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Snow Cover products from netCDF4 to GRIB2 format for EMC. • After discussions with Ken Mitchell and Michael Ek, it was determined that EMC only wants NPOESS snow products that have been passed through the IMS (Interactive Multisensor Snow and Ice Mapping System). • According to Tom Schott the Reformatting Toolkit project is not currently funded to tailor products for IMS. This is anticipated in the future although IMS may want a format other than GRIB2. • The Reformatting Toolkit development team recommends that this requirement be removed from the current phase of this project.

48. Product Requirements: Vegetation Index • Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Vegetation Index products from netCDF4 to GRIB2 format for EMC. • After discussions with Ken Mitchell and Michael Ek, it was determined that EMC only wants a Green Vegetation Fraction that is like that currently being produced from AVHRR (Le Jiang, Felix Kogan). • The NPOESS VIIRS Vegetation EDR does not have GVF. GVF was present in the VIIRS Surface Type EDR, but was removed in the last version of the Common Data Format Control Books (Feb 2009). According to Andy Heidinger, that algorithm was based on the MODIS vegetation algorithms anyway. • The Reformatting Toolkit development team recommends that this requirement be removed for the current phase of this project.

49. Interface Requirements • Requirement: The Reformatting Toolkit shall receive the thinned CrIS radiances (~300 channels) files from NUCAPS as an input for generating the CrIS radiance BUFR files. These files shall be in NetCDF4 and shall already contain the thinned SDR and geolocation information. • Requirement: The Reformatting Toolkit shall receive the NPOESS ATMS TDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the ATMS radiance BUFR files. • Requirement: The Reformatting Toolkit shall receive the NPOESS OMPS Total Column Ozone EDR files and OMPS Nadir Profile IP files tailored into NetCDF4 as an input for generating the OMPS Ozone BUFR files.

50. Interface Requirements • Requirement: The Reformatting Toolkit shall receive the NPOESS VIIRS radiance SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the VIIRS radiance BUFR files. • Requirement: The Reformatting Toolkit shall receive the NPOESS VIIRS IP Cloud Mask Product to obtain the cloud and land mask for the VIIRS radiance BUFR files. • Requirement: The Reformatting Toolkit shall receive the NPOESS Aerosol Optical Thickness EDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the Aerosol Optical Thickness EDR BUFR files. • Requirement: The Reformatting Toolkit shall receive the NPOESS SST EDR files and associated Geolocation files tailored into netCDF4 as an input for generating the SST BUFR files.