1 / 157

Prepared By: Tom King 1 , Yi Song 1 , and Kexin Zhang 1 , and Walter Wolf 2 1 IMSG

NetCDF4 Reformatting Toolkit (N4RT) Algorithm Readiness Review (ARR) for Phase 2 Products Feb 4, 2013. Prepared By: Tom King 1 , Yi Song 1 , and Kexin Zhang 1 , and Walter Wolf 2 1 IMSG 2 NOAA/NESDIS/STAR. Phase 2 ARR Objectives.

helmut
Download Presentation

Prepared By: Tom King 1 , Yi Song 1 , and Kexin Zhang 1 , and Walter Wolf 2 1 IMSG

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NetCDF4 Reformatting Toolkit (N4RT)Algorithm Readiness Review (ARR) for Phase 2 ProductsFeb 4, 2013 Prepared By: Tom King1, Yi Song1, and Kexin Zhang1, and Walter Wolf2 1 IMSG 2NOAA/NESDIS/STAR

  2. Phase 2 ARR Objectives • Demonstrate the toolkit readiness for NDE, OSPO, and data users. This requires the following: • Identify any new requirements updated since the previous ARR and show that they have been captured in the Requirement Allocation Document (RAD) and correctly implemented in the software and/or products. • Identify any bug fixes since the previous ARR. • Show the results of any new unit tests conducted since the previous ARR. • Show that the toolkit delivery items (software, documentation, test data) are present and can fully meet the requirements described in NDE DAP Document. • Review the status of all the open review items, close any remaining risks, and identify any new risks.

  3. Review Agenda

  4. Review Outline • Introduction • ARR Phase 1 Report • Updated Requirements • Software Architecture • Software Readiness • Risk Summary • Summary and Conclusions

  5. Section 1 – Introduction Presented by Tom King IMSG

  6. Project Objectives • To build a software package that will tailor JPSS and NDE products from NetCDF4 into BUFR and GRIB2 formats in support of NDE’s overall tailoring efforts. • The Reformatting Toolkit must be designed so it can easily be modified/expanded to incorporate the tailoring of new products. • Flexible • Extendable • The software must be able run within the NDE system architecture and operate within the NDE functional guidelines. • Output product formats and content must meet the needs of NOAA customers.

  7. Project Objectives Phase 1-2 Products • Phase 1 • CrIS Radiances (BUFR) • ATMS Brightness Temps (BUFR) • VIIRS Radiances (BUFR) • Sea Surface Temperature from IDPS and ACSPO (BUFR) • Phase 2 • Aerosol Optical Thickness (BUFR) • OMPS Nadir Profile (BUFR) • OMPS Total Column Ozone (BUFR)

  8. Project Objectives Phase 3-4 Products • Phase 3 • Polar Winds (BUFR) • Green Vegetation Fraction (GRIB2) • Phase 4 • OMPS Limb Profiles (BUFR) • AMSR2 TPW (BUFR) • AMSR2 CLW (BUFR) • AMSR2 SST (BUFR • AMSR2 SSW (BUFR) • AMSR2 SM (GRIB2)

  9. Project Stakeholders • IPT Lead: Walter Wolf (STAR) • PAL and IPT Backup Lead: AK Sharma (OSPO) • NESDIS team: • STAR: Walter Wolf, Jaime Daniels, Yi Song, Tom King, Kexin Zhang, Larisa Koval • NDE: Jim Silva, GeofGoodrum, Kevin Berberich, Dylan Powell, Alex Chen • OSPO: Dave Benner, AK Sharma, Ricky Irving, Zhaohui Cheng • OSD: Tom Schott • User team • Lead: Jim Heil (NWS), Stephen Lord (NWS/NCEP/EMC), John Derber and Andrew Collard (NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaardand Jim Yoe (JCSDA), Simon Elliott (EUMETSAT), Tony McNally (ECMWF), Nigel Atkinson (UK-Met) • Others: International NWP users, NWP FOs, Climate Users • Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP

  10. Customers and Users • U.S. Users: • NCEP/JCSDA (John Derber, Lars Peter-Riishojgaard) • GMAO (Will McCarty) • NRL/FNMOC (Ben Ruston) • International Users: • EUMETSAT (Simon Elliott) • UK Met Office (Nigel Atkinson) • ECMWF (Tony McNally) • DWD (Reinhold Hess) • Meteo-France (LydieLavanant) • Plus other EU member states • CMC (Louis Garand) • EC (Sylvain Heilliette) • JMA (Hidehiko Murata) • BOM (John Le Marshall)

  11. Project Timeline (1) PDR 04/28/2009 CDR 09/29/2009

  12. Project Timeline (2) TRR 10/25/2011 ARR 05/02/2012 DAP 1 Delivery 05/04/2012

  13. Project Timeline (3) DAP 2 Delivery 01/03/2013 ARR 02/04/2013

  14. Project Timeline (4)

  15. Project Timeline (5) ARR 05/06/2013 DAP 3 Delivery 05/10/2013

  16. Project Timeline (6)

  17. Project Timeline (7) ARR 10/13/2014 DAP 4 Delivery 10/17/2014

  18. Phase 2 ARR Entry Criteria • ARR Phase 1 Report (Review Item Disposition) http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT_ARR_Phase2/N4RT_Review_Item_Disposition.xlsx • PDR Risks and Actions • CDR Risks and Actions • TRR Risks and Actions • Phase 1 ARR Risks and Actions • Updated ARR Phase 1 Presentation http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT_ARR/N4RT_ARD_Phase1_SDR_Products.pptx • Updated Requirements Allocation Document http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT_ARR_Phase2/N4RT_RAD_v1.3.docx • ARR Phase 2 Presentation http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT_ARR_Phase2/N4RT_ARD_Phase2_Products.pptx • Phase 2 DAP is ready at: http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT_ARR_Phase2/ N4RT_Phase2_EDR_v1-0_20130103.tar.gz

  19. Phase 2 ARR Exit Criteria • Updated Algorithm Readiness Review Report • The ARR Report (ARRR), a standard artifact of the STAR Enterprise Process Lifecycle (EPL), will be compiled after the ARR • The report will contain: • Review Item Disposition containing all risks, actions and comments • Updated ARR presentation • Update ARD slide package

  20. Review Outline • Introduction • ARR Phase 1 Report • Updated Requirements • Software Architecture • Software Readiness • Risk Summary • Summary and Conclusions

  21. Section 2 – ARR Phase 1 Review Report: Issues and Actions Presented by Tom King IMSG

  22. ARR Phase 1 Report • This report is available as the Review Item Disposition spreadsheet at: http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT_ARR_Phase2/N4RT_Review_Item_Disposition.xlsx • The report covers the following: • Open PDR Risks and Actions • Open CDR Risks and Actions • Open TRR Risks and Actions • Open ARR Phase 1 Risks and Actions • Risks closed in previous reviews are not shown here, but are located in the RID. • Risks shown here that are marked as “closed” will be closed with the approval of this review. For “open” risks, an anticipated closure date is provided.

  23. PDR Risk • Risk 6: Risk on GVF. Have not yet demonstrated ability to encode GRIB2 files. • Assessment: Moderate • Impact: Failure to meet the requirement to have demonstrated GRIB2 tailoring capability. • Likelihood: Low • Mitigation: This work is being done now. This capability will be demonstrated in the ARR in May 2013. • Status:Open • Closure Date: May 2013

  24. TRR Risk • Risk 22: Not clear whether or not NDE has resources to finish the SPSRB documentation. • Assessment: Moderate • Impact:OSPO will not have a documented system from NDE • Likelihood:Moderate • Mitigation: Ken Jensen is working to determine what sections NDE can complete. • Status:Open • Closure Date: May 2013

  25. ARR Phase 1 Risk • Risk 26: Run times for tailoring VIIRS SDR are too large for NDE to meet their latency requirements given their current hardware resources. • Assessment: High • Impact:NCEP will not be able to receive VIIRS SDR BUFR in a timely manner. • Likelihood:High • Mitigation: Faster run times have been demonstrated by compiling and running on Linux with the GNU compiler. NDE will investigate acquisition of Linux hardware for tailoring. Full Linux transition should be completed by Jan 2014. • Status:Open • Closure Date: Jan 2014

  26. Summary • There are 3 open risks: • 1 from PDR • 1 from TRR • 1 from ARR Phase 1 • We recommend keeping all 3 risks will open at this time.

  27. Review Outline • Introduction • ARR Phase 1 Report • Updated Requirements • Software Architecture • Software Readiness • Risk Summary • Summary and Conclusions

  28. Section 3 – Updated Requirements Presented by Tom King IMSG

  29. Requirements Overview • All project requirements are documented in the RAD: http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT_ARR_Phase2/N4RT_RAD_v1.3.docx • No requirements have changed since the Phase 1 ARR so we will skip this section. The slides are here for completeness.

  30. Phase I User-to-STAR Mapping

  31. Requirement 1.0 • Requirement 1.0: STAR shall deliver to NDE a reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2).

  32. Requirement 1.0 • Requirement 1.1: The toolkit shall be capable of reformatting the NPP tailoring prioritized phase 1 product list. • Requirement 1.2: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment. • Requirement 1.2.1: The Toolkit shall compile and run on the NDE IBM AIX P5, P6, and P7 series hardware. • Requirement 1.2.2: The Toolkit shall be able interact with the NDE Data Handling System (DHS). • Requirement 1.2.2.1: The Toolkit shall be able to read a Production Control File (PCF). • Requirement 1.2.2.2: The Toolkit shall handle and return errors according to NDE/STAR standard codes. • Requirement 1.2.2.3: The Toolkit shall be able to write a (Product Status File) PSF.

  33. Requirement 1.0 • Requirement 1.3: The toolkit shall consist of modular components that can be tested independently and data should be stored in allocatable structures. • Requirement 1.3.1: The code shall consist of a single compiled program that parses arguments and logically assigns tasks to a family of hierarchically structured tailoring subroutines. • Requirement 1.4: STAR shall include one update to the reformatting toolkit within its initial project plan. • Requirement 1.5: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products. • Requirement 1.6: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit.

  34. Requirement 1.0 • Requirement 1.7: STAR shall update the reformatting toolkit when NCEP updates its BUFR and GRIB2 libraries. • Requirement 1.7.1: Updates shall be made when there are updates to the versions of the NetCDF4 library being used by NDE. • Requirement 1.8: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development. • Requirement 1.9: The output from the toolkit shall be compared with the input to verify that the conversion was performed correctly. • Requirement 1.10: The translation toolkit shall convert from the new format back into NetCDF4. • Requirement 1.11: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes.

  35. Requirement 1.0 • Requirement 1.11.1: The Reformatting Toolkit software shall generate run logs (contain human-readable messaging and time stamps) and return NDE/STAR standard (agreed upon) error codes to the DHS. • Requirement 1.12: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination). • Requirement 1.13: The toolkit parameters (e.g., how to use the service) shall be well documented. • Requirement 1.13.1: Reformatting Toolkit Developers shall provide documentation in the form of a tailored Delivered Algorithm Package (DAP) whose name and contents are defined in the NDE document entitled “Algorithm Delivery Standards, Integration, and Test V1.3”. • Requirement 1.13.2: The DAP shall contain the following two SPSRB documents: the SMM (System Maintenance Manual) and the EUM (External Users Manual) as well as the Test Readiness Document. • Requirement 14.1: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators.

  36. Requirement 1.0 • Requirement 1.14.1: Reformatting Toolkit shall use the standard set of error return codes developed by NDE for code running within the DHS. • Requirement 1.15: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting. • Requirement 1.15.1: All messaging shall be directed to a run log file. These messages shall be documented in the Reformatting Toolkit tailored DAP. • Requirement 1.16: STAR shall coordinate development of the reformatting toolkit with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system. • Requirement 1.17: Toolkit code shall adhere to the SPSRB coding standards. • Requirement 1.18: Performance shall be measured on a product level. • Requirement 1.19: The Toolkit shall output BUFR and GRIB2 files whose names adhere to the NDE file naming convention in “Algorithm Delivery Standards, Integration, and Test V1.3”.

  37. Requirement 2.0 • Requirement 2.0: STAR shall provide monthly project status reports to OSPO and OSD.

  38. Requirement 3.0 • Requirement 3.0: Earned Value Management shall be performed on the project.

  39. Requirement 4.0 • Requirement 4.0: STAR shall update the project plan on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration.

  40. Requirement 5.0 • Requirement 5.0: The toolkit shall be implemented and tested six months before the NPP launch to ensure NDE readiness.

  41. Requirement 6.0 • Requirement 6.0: The Reformatting Toolkit shall tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC.

  42. Requirement 6.0 • Requirement 6.1: The Reformatting Toolkit developers shall work with EMC and the rest of the NWP community to create a BUFR table for the NUCAPS thinned and full resolution radiances based on AIRS and IASI. • Requirement 6.2: The table shall use delayed replication for storing the radiances. • Requirement 6.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) • Requirement 6.4: The BUFR format shall allow for the storage of negative radiances. • Requirement 6.5: The file shall contain the following data fields (see table next slide):

  43. Product Requirements:CrIS Radiances

  44. Requirement 6.0 • Requirement 6.6: The reformatting Toolkit shall use the thinned CrIS radiances (399 channels) files from NUCAPS as an input for generating the CrIS radiance BUFR files for EMC. • Requirement 6.7: The reformatting Toolkit shall use the full spatial and spectral resolution CrIS radiances (1305 channels and all FOVs on all FORs) files from NUCAPS as an input for generating the CrIS radiance BUFR files for EUMETSAT.

  45. Requirement 7.0 • Requirement 7.0: The Reformatting Toolkit shall tailor the JPSS ATMS SDR and TDR data from NetCDF4 into BUFR for EMC.

  46. Requirement 7.0 • Requirement 7.1: The ATMS BUFR file shall contain, from all channels, the antenna and brightness temperatures, associated Quality Flags, and the geolocation data at native resolution (not resampled) data. • Requirement .72: The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is currently provided for AMSU and MHS. • Requirement 7.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) • Requirement 7.4: The file shall contain the following data fields (see table next slide):

  47. Product Requirements:ATMS Radiances

  48. Requirement 7.0 • Requirement 7.5: The Reformatting Toolkit shall use the JPSS ATMS TDR and SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the ATMS radiance BUFR files.

  49. Requirement 8.0 • Requirement 8.0: The Reformatting Toolkit shall tailor JPSS OMPS Ozone products from NetCDF4 into BUFR for EMC.

  50. Requirement 8.0 • Requirement 8.1: The product shall contain OMPS Nadir Profile and Total Column (multiple triplet algorithm, not v8, but output is equivalent to v8 according to Larry Flynn) in separate files. • Requirement 8.2: The Reformatting Toolkit developers shall work with EMC to develop an OMPS BUFR table based on that currently used for GOME and SBUV. • Requirement 8.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB) • Requirement 8.4: This file’s content is currently under development.

More Related