1 / 40

IDPS Block 1.5.2 AAV Verification Plan Info: CrIMSS EDR & GEO

IDPS Block 1.5.2 AAV Verification Plan Info: CrIMSS EDR & GEO. Jennifer Cram November 2 1, 2012. JPSS CGS Form J-110 10/22/2010. Objective/Agenda. Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPA at 7/10 and 8/10 meetings)

odin
Download Presentation

IDPS Block 1.5.2 AAV Verification Plan Info: CrIMSS EDR & GEO

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IDPS Block 1.5.2 AAV Verification Plan Info: CrIMSS EDR & GEO Jennifer Cram November 21, 2012 JPSS CGS Form J-110 10/22/2010

  2. Objective/Agenda • Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPA at 7/10 and 8/10 meetings) • Algorithm specific info on quality flag triggering • Algorithm specific info on IDPS QF verification plan

  3. IDPS Block 1.5.2 Qual and AAV • IDPS Block 1.5.2 Qual • Functional requirements (ING, INF, PRO, DDS, DQM) • 301 PRO functional Block 1.5.2 Qual reqts - includes ExtG, Maneuver, Graceful Deg, Interfaces, EDRIR Alg Inputs and Outputs, etc. • 246 are Test-only (Test Procedures, detailed in IDPS Test Plan) • 55 involve Analysis. Detailed in Analysis/Inspection Plans (AIPs) and Analysis/Inspection Reports (AIRs) (SE-J-198A) • Test Data sets • Transitioning to OPS 17 day • Additional specific ops data sets being collected to support verification of specific IDPS requirements for Maneuver, Solar and Lunar Eclipse, etc. • AAV Block 1.5.2- Algorithm Analysis Verification • Verification event for unique PRO EDRPR quality requirements • Requirement Re-write – are 120 reqts (47 products) • Will be AIPs and AIRs for all of these • Use OAA Test Data sets • Carefully chosen test orbits/granules to replace proxy data for algorithm chain testing (May 15, 2012 focus day) • OAA Chain runs start from RDRs

  4. IDPS EDRPR-related Requirement Verification – Past and Present • FPV – previous Block 1.0 • Verification was documented in SVRs • Verification had 3 parts • IPAC – Stand-alone and Chain comparisons to NG Truth data • Fill/Exclusion tests verified separately • Quality Flags • Proxy data sets plus 50+IDPS-altered non-nominal datasets • AAV – current Block 1.5.2 • Verification will be documented in AIPs and AIRs • Verification has 3 parts • Build-2-Build tests \(continue the baseline Pedigree from Block 1.0, SVRs, MX builds) • Exclusion/Fill Requirements verified separately • Quality Flags • OPS data sets plus minimal IDPS-altered non-nom granules (from OPS granules)

  5. Block 1.5.2 IDPS PRO EDRPR-related (AAV) Requirements • For each SDR/EDR we have a set of 3 requirements • A(Basic EDRPR functionalities): The Processing SI shall generate the xxx xDR as specified in section x.y of the JPSS Environmental Data Record (EDR) Production Report for NPP, 474-00012 • B (EDRPR Fill exclusions) : The Processing SI shall provide fill values for the xxx xDR in accordance with section x.y of the JPSS Environmental Data Record (EDR) Production Report for NPP, 474-00012 • (cloudiness exclusions , sza exclusions, etc.) • C (EDRPR Quality Flags): The Processing SI shall generate the xxx xDR Quality Flags that are listed in Table 8-x of the JPSS Environmental Data Record (EDR) Production Report for NPP, 474-00012.

  6. Approach for Verification of “A” (Basic Functionality) Reqts • Reqts with basic functionality per the EDR PR are verified by the pedigree of the B2B Quality testing • Unbroken pedigree on both the Development and Sustainment sides • Sustainment and Development B2B continuous testing • Additional testing for specific items as needed • Tests include • Specific Build-2-Build Statistical Comparisons (IDPS2IDPS) • Specific ”Calculated Truth” Comparisons (IDPS2IDPSTRUTH) • B2B Evaluation attempts to be as quantitative as practical • All comparisons, rationales documented • All artifacts CM’d • OAA B2B testing uses a standard set of IDPS Test data Granules • Transitioning from proxy to May 15 Ops set

  7. Approach for Verification of “B” (Fill Exclusion) Reqts • Fill requirements for exclusions and missing data (cloud exclusions, sza excl, etc) that are in the EDR PR. • Includes Fill Values/Types from EDR PR as well as CDFCB Definitions of Fill Values (MISS, NA, ERR, and DNE fill cases) • Will run “calculated truth” tests that fill values and associated Quality Flags are correct. • Most of these are done in build-to-build IDPS regression tests if granules trigger these behaviors • Will explicitly be analyzed in final AAV verification (AIR) • Test done in appropriate granules for the product (cloudiness, sza threshold, etc.)

  8. Approach for Verification of “C” (Quality Flag) Reqts • Quality Flag Testing • New QF’s • verified in Path C PCR S-AIRs – formal process, reviewed by NASA • Included in B2B testing from that point forward, • B2B Quality Testing: • IDPS B2B testing analyzes differences, rationale • Verifies most quality Flag's that are triggered in the B2B Test Data Granules • Non-nominal tests for specific Quality Flag's that are not triggered in the B2B Test data Granules • Limited specially-created non-nom datasets - ~ 5 separate datasets instead of the 50+ currently in use. • Limited captured special conditions datasets - (Solar, lunar eclipse, Maneuver, others) • These datasets will not trigger every single Quality Flag, but will trigger many of the most used and most important QFs • A few qf’s are not triggered at all in the nominal Test Data Granules or the NN datasets – IDPS believes that these were sufficiently verified in SVRs and do not need special datasets/runs. Identified in xls. • Detailed xls has all the info on which QF’s are verified by B2B, which require more analysis for AAV, which are not specifically verified in AAV.

  9. QF xls - Summary • QF xls was reviewed/discussed in the July 10 and Aug 10 mtgs • xls is intended to list all of the EDRPR Quality flags and to summarize the category of the planned IDPS verification for each quality flag. • How each of the quality flags was verified in the past FPV event • Whether the quality flags are triggered in the IDPS Test data sets • How IDPS/PRO is planning to verify each quality flag for AAV • The EDRPR quality flags are covered under specific Block 1.5.2 AAV IDPS/PRO requirements (under revision, 1 QF reqt per product) • Analysis Inspection Plan (AIP) is written to outline the proposed verification; the AIPs go through the critique and ECR process • Analysis Inspection Report (AIR) is written with the results, consistent with the approved AIP. The AIR also undergoes a critique and ECR process. • This is the current IDPS Plan – can be discussed, modified. But still have to balance completeness, schedule, and risk

  10. VIIRS QF Verification: VIIRS CDCNN Datasets • Carefully Designed Catastrophic Non-Nominal (CDCNN) Dataset • Careful design to remove specific VIIRS EV, Cal, and Eng/Thermal packets through a single granule to trigger as many fill and qf conditions as possible throughout the chain (xls shows a rough idea of algs/qf’s that will be affected). • Example: • M15-Band EV missing affects SDR, Imagery EDR, IST EDR, LST EDR, SST EDR • I1-Band EV missing affects SDR, Imagery EDR, SIC EDR. Snow EDRs, VI EDR • M5 Cal packet missing affects SDR qf’s, downstream cloud EDR qf’s • CDCNN usage allows efficient way to run, trigger, analyze a lot of non-nominal VIIRS behavior • CDCNN datasets will be designed for 1 tropical and 1 polar granule • Each dataset will likely have ~15 EV packet removals (15 different bands ), ~5 Cal packet removals (5 different bands), ~1 Eng packet removal. Specific scans and detectors will be carefully chosen for every packet removal in a granule dependent upon the actual scenario (ocean, land, ice, snow) and cloud cover in that granule. • Share/Use these datasets Internally and Externally

  11. Objective/Agenda • Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPE at 7/10 and 8/10 meetings) • Algorithm specific info on quality flag triggering • Algorithm specific info on IDPS QF verification plan

  12. IDPS PRO Wiring Diagram, Block 1.5

  13. Granule level QF’s – General Info • Most granule level summary qf’s are percentages: • Denominator is number of pixels that are not DNE or pixel trim • Numerator is number of pixels (per denom) triggered in a pixel-level qf. • Out of range summary qf’s have numerator and denominator only calculated over non-fill pixels • Exclusions Summary qf’s are per NG-specified specific definitions – (usually) includes all performance and production exclusions (i.e. exclusions where not produced as well as exclusions where produced but under conditions that are not supposed to count towards performance reqts (in original NPOESS Sys Spec).

  14. CrIMSS Info • CrIMSS EDR and CrIS IR Ozone IP • 4 scans, 30 FOR/scan • 22 layers AVMP, 42 layers AVTP, 31 layers AVPP • CrIS IR Ozone IP • 4 scans, 30 FOR/scan • 101 OSS layers (1100 - .005 mb) • From CDFCB: only CrIS Infrared radiances are used in the forward model to obtain ozone - ATMS microwave radiances are not used. However, some parameters retrieved from the first stage microwave retrieval (ATMS) are used when available within the ozone retrieval. Therefore, the ATMS retrieval related fields are included as part of this product.

  15. CrIMSS Info • 1 CrIMSS retrieval per FOR – uses all 9 CrIS FOV • AVMP (water vapor) – 22 layers at specified pressure levels, 2 km vertical cell average (cells overlap). Value is average value over that cell. • AVTP (temperature) – 42 layers at specified pressure levels, vertical cells range from 1-5 km. Value is average value over that cell. • AVPP (Pressure on Height levels) – 31 height layers, vertical cell size 2 km • Block 2 originally was supposed to have 1/4/9 retrievals/FOR – that functionality/reqts are being removed

  16. CrIMSS Info • CrIMSS AVTP and AVMP fields are NA fill when cell is partially or completely below terrain • complex because cell encompasses many levels • Note from discussion: surface pressure product is terrain-corrected pressure. Surface Pressure is based on average terrain in the FOR. This has effect when high point of the terrain (that went into the average) is above the next pressure level up. • Results would show fill layer even though surface pressure level below that. Will be more likely to see this where • a) Significant terrain differences – high variance within FOR • b) where surface pressure is close to (just under) a specified pressure layer (so occasional points could stick up)

  17. CrIMSS Info – SDR Inputs • Inputs from SDRs: • CRIS SDR granule is radiances at 4 granules x 30 FOR x 9 FOV x 3 bands • ATMS SDR granule is Brightness Temps at 12 scans x 22 channels x 96 beam positions • ATMS Remap is a remap of the ATMS SDR to the CrIS Geolocation (CrIMSS res 4 x 30 FOR x 22 channels) • Time is actually used for the remap (not Lat/lon) • Remap used Backus-Gilbert resampling to resample the ATMS SDR to the CrIS SDR geolocation • When the CrIS GEO is missing (Cris SDR was not created for some reason) then a “fake Cris GEO” is created in the code by assuming nadir pointing

  18. CrIMSS Info – Missing Data • MISS fill when both ATMS Remap and CrIS have MISS fill (overlapping Data Gaps). • If both SDR granules are completely missing then the CRIMSS EDR will not be produced • ATMS-only missing(gap or entire granule) • CrIMSS still produced • ATMS info comes from Gran-ANC (Water vapor and Temp profiles) • CrIMSS product is marked as gracefully degraded • QF’s triggered: • ATMS available is “Not available” • Overall Quality is IR-only • CrIS-only missing (gap or entire granule) • CrIMSS still produced (ATMS Remap “fake CrIS GEO”) • QF’s triggered: • ATMS available is “Available” • Overall Quality is MW-only • CrIS Ozone IP is MISS fill when CrIS SDR is MISS fill

  19. CrIMSS and CrIS Ozone IP Fill tests for Block 1.5.2 • CrIMSS EDR is NA fill in H20 and Temperature fields when the pressure layer is partially below the terrain. • CrIMSS EDR is NA fill in H20 and Temperature fields when the pressure layer is totally below the terrain. • CrIMSS EDR is MISS fill in Water Vapor, Pressure, and Temperature fields if CrIS and ATMS SDR inputs are missing. • CrIS IR Ozone Profile IP O3 field is MISS fill if CrIS SDR is set to MISS fill.

  20. CrIMSS EDR QF’s • Granule-level • AVTP Product Yield (same as AVMP, AVPP) • CrIS Input Data Quality • ATMS Input Data Quality • CrIS SDR detector failure (9 det x 3 bands) • Apodization Flag • Day/Night flag (granule level)

  21. CrIMSS EDR QF’s • FOR-level • Ice mask • AVTP overall quality (same as AVMP, AVPP) • ATMS Input SDR Quality (22 channels) • CrIS Input SDR Quality (3 bands x 9 FOV) • ATMS available • Convergence flag for combined IR-MW retrieval • Convergence flag for MW only retrieval • Temperature Difference • Clear/Cloudy Flag • Rain flag • Day/Night flag • Sun glint flag • Coast flag (output as land frac “field”) • Cell Size • Temperature out of range flag • Non-LTE Flag • Chi sq value from joint IR-microwave radiance matching • Chi sq value from microwave radiance matching - Stage 1 • Chi sq value from microwave radiance matching - Stage 2 • IR noise amplification factor

  22. CrIMSS EDR Granule-level QF’s • AVTP Product Yield (same as AVMP, AVPP) • granule-level QF that summarizes the number of high quality retrievals in a granule. The numerator is the number of retrievals where the retrieval-level Overall Retrieval Quality QF (QF5, bits 3 and 4) is set to High (IR+MW) (value of 0). The denominator is the number of retrievals in the granule. VDNE fill, which occurs in short granules, is not included in the numerator or denominator counts. • CrIS Input Data Quality • percent of retrievals where the FOR-level qf “CrIS SDR Quality” has a value of “Good” for all 3 bands for all 9 FOVs in the FOR. The denominator is the number of retrievals in the granule, which excludes retrievals that have DNE fill (short granule). • ATMS Input Data Quality • percent of retrievals where the CrIMSS “ATMS SDR Quality” QF has a value of “Good” for all 22 ATMS channels. The denominator is the number of retrievals in the granule, which excludes retrievals that have DNE fill (short granule).

  23. CrIMSS EDR Granule-level QF’s • CrIS SDR detector failure (9 det x 3 bands) • set when the “detectorQF” array in the CrIMSS PCT file (CrIMSS-EDR-AC) indicates a failed CrIS detector, or when the CrIS SDR is not present. • Note that the CrIS SDR PCT file (CrIS-SDR-CC) contains a separate array “detector” that also contains the information about bad CrIS detectors. When a detector is denoted as bad in the CrIS-SDR-CC file, the CrIS SDR code does not process interferogram data for that band / FOV. • If a CrIS detector is bad, data should be updated in both the CrIMSS EDR PCT file and in the CrIS SDR PCT file. The array in the CrIS SDR PCT file is a two-dimensional array, while the CrIMSS EDR PCT file array is a one-dimensional array, so the location of the bit for a specific detector is different in the two arrays.

  24. CrIMSS EDR Granule-level QF’s • Apodization Flag • value of the apodFlag parameter in the CrIMSS PCT file. The code is using “Blackman-Harris”. • Day/Night flag (granule level) • 0: Solar Zenith Angle is less than 90 degrees for all CrIS FOVs within the granule; • 1: Solar Zenith Angle is equal to or larger than 90 degrees for all CrIS FOVs within the granule; • 2: None of the above.

  25. CrIMSS EDR FOR-level QF’s • AVTP/AVMP/AVPP overall quality • Note that may still have an output retrieval even if Stage 1 and 2 do not converge

  26. CrIMSS EDR FOR-level QF’s • Ice mask • Set when nonFILL MW-only surface temperatures (i.e., skin temperatures) are below freezing (273.15 degrees K) over land pixels. • Note that when the stage 2 retrieval (IR+MW) converges, then the CrIMSS EDR skin temperature field is set from the stage 2 retrieval results, while the Ice Mask QF is set based on the stage 1 (MW only) retrieval results. • ATMS Input SDR Quality (22 channels) • Pass-thru from ATMS Remap Resampling QF, for each of 22 channels • CrIS Input SDR Quality (3 bands x 9 FOV) • Pass-thru from CrIS SDR “SDR Quality” QF,for each of 9 Fields of View (FOVs) and for each of 3 bands (SW, MW, LW).

  27. CrIMSS EDR FOR-level QF’s • ATMS available • If ATMS data is “Available”, the QF will have a value of “False” (0). If ATMS data is not available, the QF will have a value of “True” (1). • In cases where some but not all channels are missing, the QF indicates that ATMS is available. • For a particular FOR and channel, the ATMS REMAP algorithm uses a set number of brightness temperatures from a set of beam positions determined by a LUT. Some channels in FORs near the data gap require brightness temperatures from beam positions that are missing due to the data gap. In these cases, if the Backus-Gilbert coefficients are below a certain threshold, the ATMS REMAP algorithm will not be able to determine a valid brightness temperature and that channel/FOR will be set to MISS fill.

  28. CrIMSS EDR FOR-level QF’s • Convergence flag for combined IR-MW retrieval • Stage 2 convergence • No if ATMS missing • No if Stage 1 converged, but CrIS missing • Convergence flag for MW only retrieval • Stage 1 (ATMS) convergence • Temperature Difference • set when the Root Mean Square (RMS) of the difference between the temperatures from the combined (MW+IR) retrieval and the temperatures from the microwave-only (MW) retrieval in the lowest 6 pressure levels above the topography and 1 pressure level below the topography, for a total of 7 pressure levels, exceeds a configurable value. The default configurable value is 3.0. • Temperatures from the combined (MW+IR) retrieval are found in the CrIMSS-CrIS-AVTP-LOS-IR-IP. Temperatures from the microwave-only (MW) retrieval are found in the CrIMSS-CrIS-AVTP-LOS-MW-IP.

  29. CrIMSS EDR FOR-level QF’s • Clear/Cloudy Flag • set by checking the values of two variables (clrFlg(1) and ncldfm) which are computed in the subroutine Fovsel.f: • Clear (set flag to zero) if ncldfm = 0 and clrFlg(1) = .true. • Partly cloudy (set flag to 1) if 0<ncldfm<=3 • Cloudy (set flag to 2) if (ncldfm = 0 and clrFlg(1) = .false. ) or (ncldfm >=4) • Note that the implementation of the CrIMSS Clear/Cloudy QF and the underlying CrIMSS cloud detection algorithm are very different than that of VIIRS CM (VCM cloudiness is not “similar” to CrIMSS cloudiness) • Day/Night flag • set when the CrIS SDR GEO Solar Zenith Angle for the middle FOV (5th index in the HDF5) in the FOR is greater than or equal to 90 degrees. • Sun glint flag • The sun glint condition occurs when the angle formed by the sun and the sensor’s specular direction equals or is less than 36° (tunable threshold). • sensor angles of the center FOV should be used in the calculations.

  30. CrIMSS EDR FOR-level QF’s • Rain flag • EDRPR description: Indicates precipitation within the FOR exceeding 2 mm/hr • Coast flag • Not in a QF byte – same as the Land Frac product (0-1) • Cell Size • Values: • 0 = 9 FOVs used • 1 = 4 FOVs used • 2 = 1 FOV used • 3 = No Retrieval • Only 0 and 3 are triggered through Block 1.5.2 • Non-LTE Flag • triggered when the local solar zenith angle is equal to or less than 90° (i.e., day time).

  31. CrIMSS EDR FOR-level QF’s • Temperature out of range flag • Set when the CrIMSS EDR skin temperature or any CrIMSS-CrIS-AVTP-LOS-IR-IP tempProfile value is below the minTemp value or above the maxTemp value from the CrIMSS PCT file for any retrieval iteration. • Note that the Temperature Out of Range QF could be triggered during an early iteration, even though the final iteration values of skin temperature and tempProfile are not out of the expected range. • Chi sq value from joint IR-microwave radiance matching • In “product fields, not QF byte • Chi sq value from microwave radiance matching - Stage 1 • In “product fields, not QF byte • Chi sq value from microwave radiance matching - Stage 2 • In “product fields, not QF byte • IR noise amplification factor • In “product fields, not QF byte

  32. CrIS IR Ozone IP QF’s • Granule-level that are same as CrIMSS EDR QF’s: • CrIS Input Data Quality • ATMS Input Data Quality • CrIS SDR detector failure (9 det x 3 bands) • Apodization Flag • Day/Night flag (granule level) • Granule-level that are different from CrIMSS EDR QF’s: • Ozone Profile Product Yield • Percentage of high quality retrievals in a granule. The numerator is the number of retrievals where the retrieval-level CrIS IR Ozone Profile IP “Overall Retrieval Quality” QF (QF5_CrISIRO3, bits 3 and 4) is set to High (IR+MW) (value of 0). The denominator is the number of retrievals in the granule. • VDNE fill, which occurs in short granules, is not included in the numerator or denominator counts.

  33. CrIS IR Ozone IP QF’s • FOR-level that are same as CrIMSS EDR QFs • Ice mask • ATMS Input SDR Quality (22 channels) • CrIS Input SDR Quality (3 bands x 9 FOV) • ATMS available • Convergence flag for combined IR-MW retrieval • Convergence flag for MW only retrieval • Temperature Difference • Clear/Cloudy Flag • Rain flag • Day/Night flag • Sun glint flag • Coast flag (output as land frac “field”) • Cell Size • Temperature out of range flag • Non-LTE Flag • Chi sq value from joint IR-microwave radiance matching • Chi sq value from microwave radiance matching - Stage 1 • Chi sq value from microwave radiance matching - Stage 2 • IR noise amplification factor

  34. CrIS IR Ozone IP QF’s • FOR-level that are different from CrIMSS EDR QFs • IR Ozone overall quality

  35. CrIS IR Ozone IP QF’s • FOR-level that are different from CrIMSS EDR QFs • Ozone Spectral Signature • In “product” field, not QF byte • defined in TM NP-END.2009.510.0011.Rev.A • calculated from the radiance values in the CrIMSS-CrIS-CLOUD-CLEARED-RAD-IP “R(n)”as follows: Ozone Spectral Signature = (R(529)+R(673) – R(625) – R(646) )/2.

  36. CrIMSS EDR GEO QF • Missing Ephemeris or Attitude Data • S/C Diary RDR Data (20 sec granules, 1 packet/sec) • CGS also has daily TLE, SGP4 propagation • QF Options • Nominal • Small Gap (<= 30 secs) • Ephemeris and attitude interpolated between ends • Large Gap (greater than small gap) or overlaps Granule Boundary • Ephemeris uses prediction from TLE, interpolation between prediction and S/C diary ephemeris. • Attitude assumed perfect nadir • Missing for Entire granule (Uses Prediction from TLE) • Ephemeris uses prediction from TLE • Attitude assumed perfect nadir

  37. Objective/Agenda • Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPE at 7/10 and 8/10 meetings) • Algorithm specific info on quality flag triggering • Algorithm specific info on IDPS QF verification plan

  38. CrIMSS EDR QF Verification Summary • Ops data set (15 May) has not yet been analyzed for CrIS, ATMS, CrIMSS triggers. • We are planning to “make” or find Non-nominal datasets for: • CrIS and ATMS Data Gap (each separately as well as overlap) • Missing Ephemeris

  39. CrIMSS EDR QF Verification • Review QF xls: • CrIMSS EDR QF’s • CrIMSS EDR GEO QF’s • CrIMSS Ozone IP QF’s

  40. References • FPV SVRs –Erooms at Ground Data Product Engineering > IDPS SVRs • IDPS AAV Info – Erooms at Ground Data Product Engineering > IDPS AAV Verification

More Related