1 / 60

Section 7 – Quality Assurance Presented by Zhaohui Cheng PSGS

Section 7 – Quality Assurance Presented by Zhaohui Cheng PSGS. Quality Assurance. Quality assurance consists of PROCESS QA and PRODUCT QA

benny
Download Presentation

Section 7 – Quality Assurance Presented by Zhaohui Cheng PSGS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Section 7 – Quality Assurance Presented byZhaohui Cheng PSGS

  2. Quality Assurance • Quality assurance consists of PROCESS QA and PRODUCT QA • PROCESS QA is concerned with assuring that the STAR EPL process standards are met during the planning, development, operations, and distribution phases of the project. • Process QA is achieved through the standard reviews of the STAR EPL process. Each review checklist and entry/exit criteria are designed to ensure that the relevant process standards are met by the implementation of standard practices during the steps leading up to the review. • PRODUCT QA is concerned with assuring that the work products created during the project’s lifecycle meet their requirements • Product QA is achieved by verification of the project’s work products and validation of the products, operator needs, and user needs

  3. Configuration Management Stakeholders • Xingpin Liu • CM/DM Lead • Yunhui Zhao • CM Administrator • Mark Romer, Brian Keffer (STAR IT) • CM installation • Tom King, Chen Zhang, Zhaohui Cheng, Antonia Gambacorta, Haibing Sun and Kexin Zhang (Develop team members) • CM Users

  4. Configuration Management Tools • CM Tool (IBM Rational ClearCase ) • has been purchased and implemented in STAR Collaborate Environment • CM Administrators: • Xingpin Liu • Yunhui Zhao • CM training: • Administrator training completed (Xingpin Liu) • IBM on site training for developers is being scheduled

  5. Project Baseline Report • The project’s baseline and change history is maintained in a Project Baseline Report (PBR). • Document guidelines are in STAR EPL process asset DG-9.6. • <Pointer to DG-9.6> • The PBR includes the change history, approval status, and location of every Configuration Item in the project’s baseline. • PBR v1r0, a CDR artifact, can be accessed at <pointer to PBR v1r0>

  6. Verification and Validation Plan • Verification is the formal process of confirming that the requirements specified for a specific product or system are satisfied by the completed product or system. • Validation is a process of evaluation, integration, and test activities conducted to ensure that the final developed system will satisfy the needs and expectations of customers, users, and operators • In a well-designed system, needs and expectations are completely captured by the requirements allocation (see Section 8 of this CDD) – in that case, there is no meaningful distinction between verification and validation • The methods and planned activities for verification and validation of the project’s process and products constitutes the project verification and validation plan

  7. Verification and Validation Documentation • The plan for verification and validation is documented in a Verification and Validation Plan (VVP). It is a review artifact for this CDR. • When the first phase of development is complete, a Unit Test Plan (UTP) will be developed and presented at a Test Readiness Review (TRR). • After the TRR, a Code Unit Test Review (CUTR) will be conducted to demonstrate that code meets the requirements. • Prior to delivery, a Validation and Verification Report (VVR) will be generated to show that the results of the tests in the VVP and UTP were successful and that the system will be able to meet the needs of users identified in the RAD. • VVP v1r0, a CDR artifact, is available at http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/NUCAPS_CDR/NUCAPS_VVP_1.0.doc • Documents the refinement of the plan for a preferred solution and RAD v1r0

  8. Verification Methods • Analysis consists of activities such as the generation of statistics to examine the distribution and errors and the comparison of results with reference data. • Inspection is simply verifying that required element is present or that the expected output was obtained. • Demonstration is meant to illustrate the functionality of something such as a software unit.

  9. Verification― Data Product

  10. Verification― Software and Tool Item Method

  11. Verification― Documentation Item Method

  12. Validation of Products―Thinned Radiances • The thinned radiance products generated by the NUCAPS software are the granule CrIS Sensor Data Record (SDR) data that have been apodized and spatially thinned to reduce the coverage and spectrally thinned to reduce the number of channels. • Requirement • Basic Requirement 1.0 • Derived Requirement 1.1 • Derived Requirement 1.1.1 • Derived Requirement 1.1.3 • Derived Requirement 1.1.4 • Derived Requirement 1.1.2.6.1 • Derived Requirement 1.2 • Derived Requirement 1.2.1 • Derived Requirement 1.3 • Derived Requirement 1.3.1

  13. Validation of Products―Thinned Radiances • The thinning verification will run the Subsetter unit and then compare the input full resolution file to the output thinned data files. • To test BUFR file, dumps of the NetCDF files that were generated from the BUFR files indicated that: • All FOVs in the file were completely filled with data (file was completely read) • Content of the output NetCDF files matched that of the input NetCDF files. • To test the warmest FOV selection, the radiance values for the window channel used in the selection will be dumped along with the footprint index to show that the correct warmest FOV within the FOR was chosen.

  14. Validation of Products―Thinned Radiances • Requirement 1.1.1 states that these radiances must be apodized (Hamming apodization). This processing will be performed in the Preprocessor unit. • Hamming apodization must be reversible. • Apodized spectra will be compared to simulated CrIS spectra at the same locations/times. If done correctly ringing should not be present in the apodized CrIS spectra.

  15. Validation of Products―Thinned Radiances • Requirement 1.1.3 is the timeliness requirement for the thinned radiance products for NCEP. • There are problems with verifying this requirement, we will discuss it in the risk section. • The only way to verify this requirement is to ingest actual data after launch, compare the time of receipt to the observation time, add that to the amount of NUCAPS processing time, and then estimate how much longer it will take for the NDE DHS to pass the product to the customers. The sum of those times compared to the three hour window specified in the requirement should allow the development team to know if the requirement is being met.

  16. Validation of Products―Thinned Radiances • Requirement 1.1.2.6.1 states that the VIIRS FOVs must be collocated to the CrIS FOVs. Tests will be performed to see if the collocation was successful. • An output netCDF4 granule from the Preprocessor unit will be selected containing a variety of surface and cloud variability • The CrIS channels will be convolved to the VIIRS channels so CrIS and VIIRS are on the same spatial reference. • Averaged over the whole granule, the spectrally convolved CrIS radiances should match those of the spatial averaged VIIRS.

  17. Validation of Products―Principal Components • The NUCAPS principal component (PCS) products are generated from the CrIS SDR data with and one or more eigenvector coefficient files. • Requirement • Basic Requirement 2.0 • Derived Requirement 2.1 • Derived Requirement 2.1.1 • Derived Requirement 2.1.2 • Derived Requirement 2.2 • Derived Requirement 2.2.1 • Derived Requirement 2.2.2 • Derived Requirement 2.2.2.1

  18. Validation of Products―Principal Components • To validate the eigenvector files: • Plot eigenvalues for the training day • The number of pieces of information is determined by the “knee” • The “knee” is transition from signal to noise floor • Random noise should generate constant eigenvalues

  19. Validation of Products―Principal Components • Simulated CrIS data on July 18, 2008 (eigenvalues as function of index) knee

  20. Validation of Products―Principal Components To validate the principal components: • Compute reduced noise radiances (reconstructed radiances) using PC. • Subtracting observed radiance from reconstructed radiances gives an estimate of instrument noise derived from Earth scenes. • Check if the estimate noise is close to black body derived noise (NEDT)

  21. Validation of Products―Principal Components • Compare IASI reconstruct radiances with observed radiances at 2500cm-1 upper left: observation upper right: reconstructed lower left: obs – rec. lower right: histogram of diff

  22. Validation of Products―Principal Components • At upper panel is the NEDT noise estimate for a single channel (2500 cm-1 on Sept. 10, 2007) shows the expected characteristics as a function of scene temperature (red lines are 1 sigma NEDT and green is 2 sigma NEDT). • At lower panel is IASI noise (red curve) derived from blackbody measurements compared with noise derived from PC’s. PC’s generated from all 8461 channels shown in blue) and PC’s generated from the 3 individual bands (green) are very similar and very close to the black body derived noise.

  23. Validation of Products―Principal Components • Principal Component (PC) Scores are a quick diagnostic of instrument stability and performance. • We use a couple of days of instrument radiances to compute a radiance covariance for CrIS. • Using the eigenvectors of the covariance we can monitor that CrIS radiances are behaving as expected, statistically.

  24. Validation of Products―Principal Components • Typically PC Scores range from 0.7 to 1.3. Instrument problems (or rare geophysical events like volcanoes) show up as large PC scores. • Statistics of global daily PC Scores enable quick monitoring of instrument health.

  25. Validation of Products―NUCAPS EDR • The NUCAPS retrieval products are single granule files that contain profiles of trace gases as well as some surface and cloud radiative properties. They are generated from the granule CrIS and ATMS SDR data with the NCEP Global Forecast System (GFS) six-hour forecast surface pressure. • Requirement • Basic Requirement 3.0 • Derived Requirement 3.1 • Derived Requirement 3.1.1 • Derived Requirement 3.1.2 • Derived Requirement 3.1.3 • Derived Requirement 3.2 • Derived Requirement 3.2.1 • Derived Requirement 3.2.2 • Derived Requirement 3.2.3 • Derived Requirement 3.3 • Derived Requirement 3.3.1 • Derived Requirement 3.5 • Derived Requirement 3.6

  26. Validation of Products―NUCAPS EDR(Temperature and Moisture) • Trace gas profiles and cloud clear radiances are depend on the temperature and moisture profiles • The temperature and moisture profiles can be used to validate operational products. • NUCAPS EDR are the heritage of AIRS and IASI to build a 20 years climatology products. • The verification of the profiles product will be done by comparing the NUCAPS temperature and water vapor retrievals with models analysis field (ECMWF). • Check if the temperature and moisture profiles are reasonable • Provide a quick look of global skill and can be used to identify problematic regions

  27. Validation of Products―Retrieval Products (Temperature and Moisture) • Example to the right is a comparison of IASI level 2 products with ECMWF analysis fields at ~400mb • Care must be taken to separate model issues, such as time of day, location of fronts, etc. • Similar displays of instrument radiances (raw or cloud cleared) and model fields (potential vorticity, etc.) enable tracing EDR issues back to their source.

  28. Validation of Products―NUCAPS EDR (Temperature and Moisture) • The validation of the profiles product also can be done by comparing the NUCAPS temperature and water vapor retrievals to the profiles produced by collocated radiosonde measurements. • The NUCAPS retrievals will be matched to radiosondewithin ± 3 hours and within 100 km • Temperature statistics will be generated for 1 km layers from surface to 100mb. For water vapor, statistics will be computed for column densities converted to integrated column water in 2km layers from the surface to 100 mb.

  29. IASI- Temperature and WV RMS Difference – Sea(Compare with Radiosonde) RAOB vs. IASIAVNATOVSECMWFFG

  30. The Validations by Comparing with Radiosonde

  31. Validation of Products―NUCAPS EDR (Temperature and Moisture) • Compare temperature and water vapor retrieval withdedicated sondes [ARM sites: SGP (97.5W, 36.6N), TWP (166.9E, 0.5S) , NSA (156.6W,71.3N)]

  32. Validation of Products―NUCAPS EDR (Trace Gases) • Have the experience of 6+ years of AIRS and ~2 years of IASI retrieval processing. AIRS products had been well validated and IASI validation is ongoing. • The NUCAPS trace gases profiles can be indirectly validated by being compared with profile products from AIRS and IASI. This is the quick way to check the consistency and reasonability of NUCAPS EDR.

  33. Comparison of IASI and AIRS Ozone Product, Oct. 19, 2007 IASI 21:30 Ascending (night) AIRS 13:30 Ascending (day)

  34. IASI & AIRS Carbon Monoxide 2 1 3 4

  35. Validation of Products―NUCAPS EDR (Trace Gases) • Another validation activity will be to interact with in-situ measurement campaigns during the post launch phase of development. Will need years to collect the ground truth values. • ESPL continuity • Hopefully IPO will fund the validation • Participate any scientific campaigns and opportunities (“START”, “HIPPO” and “INDEX”) • The validation team will provide the retrieval products and get the in-situ data in return • Exploit scientific collaboration to provide science community to exchange low cost validation

  36. Stratospheric-Tropospheric Analysis of Regional Transport (START) Experiment • Laura Pan is PI of START Ozone team • Nov. 21 to Dec. 23, 2005, 48 flight hours using NCAR’s new Golfstream V “HAIPER” aircraft. • Ozone measured with NCAR’s UV-abs spectrometer • NOAA NESDIS supported this experiment with real time AIRS L1b & L2 products, including ozone and carbon monoxide. • Received a letter of appreciation from the head of NCAR for our collaboration with this experiment. • Jennifer Wei is the NOAA/NESDIS liason to START team. • 3 stratospheric fold events were measured during this campaign • analysis is in process.

  37. Support of START08 and Pre-HiPPO – Use of AIRS and IASI to study Transport and Vertical Mixing http://catalog.eol.ucar.edu/start08/index.html Figure 1 • We participated in the NCAR Stratosphere-Troposphere Analyses of Regional Transport 2008 (START08) and Harvard Preliminary HAIPER Pole-to-Pole Observation (pre-HIPPO) experiments from April to June, 2008. • We provided near real time level-2 products derived from the AIRS and pre-operational products from IASI. Satellite derived tropopause height, H2O, O3, CO and CH4 were used for daily flight forecast. AIRS and IASI have demonstrated to be great instruments for “chemical tracers predictions” over the continental US region. • Figure 1: IASI derived ozone (O3) at 200 mb shows patterns consistent with upper tropospheric dynamics (stratospheric intrusions, red contours) ; • Figure 2: IASI carbon monoxide (CO) at 500 mb shows high CO over Oregon/Idaho due to long range transport of recent Russian fire. Figure 2

  38. Methane validation: comparison w/ ESRL aircraft • Compared AIRS methane product to a number of ESRL aircraft sites. • The accuracy of AIRS CH4 is about 1.2-1.5% depending on altitude. Able to map seasonal variation of CH4 and can provide valuable information in mid-upper troposphere.

  39. AIRS CH4 vs Aircraft Measurement – 5 years data(200, 300, 400, 650 mb ) ESRL/GMD CH4 aircraft profiles are the best validation for thermal sounders since they measure a thick atmospheric layer. Collocation ∆R < 200 km ∆t < 24 hour

  40. Direct Comparison: AIRS CO2 versus ESRL Aircraft • Comparison of AIRS retrieved CO2 shows improvement in statistics (dashed vs. solid lines) as a function of temporal averaging (e.g., Harvard Forest site, upper right) up to about 5 days of averaging. • Averaging improves signal to noise • Comparisons between “point” aircraft and “volume” satellite measurements are more difficult. • Validation scatter plot (lower right) for all ESRL/GMD sites for the largest collocation time window (+/- 14 days) shows we are within our expectation of 2 ppmv (0.5%). • Tendency to miss large draw-down events – appears to be both a retrieval issue and in-situ sampling.

  41. Trace GAS web-pages http://www.orbit.nesdis.noaa.gov/smcd/spb/iosspdt/testd • Trace GAS web-pages allow a quick look at the trace gas products as a function of geography, time, and comparisons with in-situ datasets. USERID & PASSWORD Request via e-mail: chris.barnet@noaa.gov

  42. Validation of Products―Cloud-Cleared Radiances • The Cloud-Clear Radiance (CCR) is NOAA unique product to be generated by the retrieval algorithm in the Retrieval unit. • Requirement • Basic Requirement 4.0 • Derived Requirement 4.1 • Derived Requirement 4.2 • Derived Requirement 4.2.1 • Derived Requirement 4.2.2 • Derived Requirement 4.2.3 • Derived Requirement 4.2.4

  43. Validation of Products―Cloud-Cleared Radiances • Validate CCR with simulated clear radiance • The validation of temperature and moisture profiles with radiosonde is the indirect way to validate the CCR • Validate CCR by comparison to other instrument observations. • The validation team will collocate VIIRS observations to CrIS CCR FOVs and then spectrally convolve the CrIS CCR channels to those of VIIRS as well as spatially convolve clear-masked VIIRS radiances to the CrIS FOVs. • CrIS CCR should match closely with the clear VIIRS radiances.

  44. Validation of Products―Cloud-Cleared Radiances • Compare AIRS CCR radiances with those derived from numerical model simulations of clear conditions.

  45. Validation of Products―Cloud-Cleared Radiances • An example was validating the AIRS CCR product with MODIS data Cloud-Cleared AIRS Spectrally convolved to MODIS MODIS Clear-masked Spatial Convolved to AIRS FOV

  46. Validation of Products―Cloud-Cleared Radiances • Verification of the CCR product will satisfy Requirements 4.0, 4.1, and 4.2. Requirement 4.2.1 states that the NUCAPS CCR product will be delivered in netCDF4 format. • Running the Retrieval unit to show that the netCDF4 file can be successfully generated. Then, the file’s contents will be dumped to show that it contains reasonable, non-missing values. • Verify the metadata for the CCR • running the Retrieval unit to demonstrate the software’s functionality • dump the metadata file contents and running them through the USGS mp metadata checking tool. • The CCR metadata will be validated by sending it to Anna Milan at the NGDC

  47. Validation of Products―Daily Global Products • There are two types of daily global products: (1) matchups and (2) grids. Both of these products are generated within the Validation unit • The matchups read in a day’s worth of NUCAPS data then collocate those observations nearest in space and time to in-situ instrument observations. • Global grids are daily globally gridded product files.Grids are available in 0.5X2.0 and 3.0X3.0 degree resolution.

  48. Validation of Products―Daily Global Products • Requirement • Basic Requirement 5.0 • Derived Requirement 5.1 • Derived Requirement 5.1.1 • Derived Requirement 5.1.1.1 • Derived Requirement 5.1.1.1 • Derived Requirement 5.1.2 • Derived Requirement 5.1.2.1 • Derived Requirement 5.1.2.1 • Derived Requirement 5.2 • Derived Requirement 5.2.1 • Derived Requirement 5.2.1.1 Derived Requirement 5.2.1.1 • Derived Requirement 5.2.1.2 • Derived Requirement 5.2.2 • Derived Requirement 5.2.2.1 • Derived Requirement 5.2.1.1 • Derived Requirement 5.2.2.2 • Derived Requirement 5.2.3 • Derived Requirement 5.2.3.1 • Derived Requirement 5.2.3.1 • Derived Requirement 5.2.3.2 • Derived Requirement 5.2.4 • Derived Requirement 5.2.4.1 • Derived Requirement 5.2.4.1.1 • Derived Requirement 5.2.4.2

  49. Validation of Products―Daily Global Products • To verify the functionally of the matchup code • print out the latitude/longitude and time information for a given in-situ measurement. • dump out all the satellite measurement locations/time that met that minimum requirement. • As the program runs through all the satellite data for a given match to an in-situ location (+/- 3 hours and 100 km), the matches should converge to the nearest observation. • The same test will verify the functionality of both the radiance and retrieval matchups. Both sets of matchups should also match to the same set of in-situ measurement points. • This is also easily verified by running retrievals on the radiance matchups. The retrievals generated from the radiance matchups should match, identically, those of the NUCAPS retrieval matchups.

  50. Validation of Products―Daily Global Products • A test will be conducted to demonstrate that the satellite observations can be matched to the grid points. • Write out a given grid point’s latitude and longitude. • All the observations that mapped to that point will be printed out. As with the matchups, they should converge to the nearest satellite observation if the gridding is being done correctly. • An additional method for verifying the content of these grids will be to give them to the user (the NUCAPS V&V team) for validation and verification. • Plot gridded radiances for ascending and descending orbits for a window channel using GrADS

More Related