390 likes | 555 Views
IDPS Block 1.5.2 AAV Verification Plan CrIS SDR and GEO Info. Jennifer Cram November 7 , 2012. JPSS CGS Form J-110 10/22/2010. Objective/Agenda. Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPA at 7/10 and 8/10 meetings)
E N D
IDPS Block 1.5.2 AAV Verification PlanCrIS SDR and GEO Info Jennifer Cram November 7, 2012 JPSS CGS Form J-110 10/22/2010
Objective/Agenda • Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPA at 7/10 and 8/10 meetings) • Algorithm specific info on quality flag triggering • Algorithm specific info on IDPS QF verification plan
IDPS Block 1.5.2 Qual and AAV • IDPS Block 1.5.2 Qual • Functional requirements (ING, INF, PRO, DDS, DQM) • 301 PRO functional Block 1.5.2 Qual reqts - includes ExtG, Maneuver, Graceful Deg, Interfaces, EDRIR Alg Inputs and Outputs, etc. • 246 are Test-only (Test Procedures, detailed in IDPS Test Plan) • 55 involve Analysis. Detailed in Analysis/Inspection Plans (AIPs) and Analysis/Inspection Reports (AIRs) (SE-J-198A) • Test Data sets • Transitioning to OPS 17 day • Additional specific ops data sets being collected to support verification of specific IDPS requirements for Maneuver, Solar and Lunar Eclipse, etc. • AAV Block 1.5.2- Algorithm Analysis Verification • Verification event for unique PRO EDRPR quality requirements • Requirement Re-write – are 120 reqts (47 products) • Will be AIPs and AIRs for all of these • Use OAA Test Data sets • Carefully chosen test orbits/granules to replace proxy data for algorithm chain testing (May 15, 2012 focus day) • OAA Chain runs start from RDRs
IDPS EDRPR-related Requirement Verification – Past and Present • FPV – previous Block 1.0 • Verification was documented in SVRs • Verification had 3 parts • IPAC – Stand-alone and Chain comparisons to NG Truth data • Fill/Exclusion tests verified separately • Quality Flags • Proxy data sets plus 50+IDPS-altered non-nominal datasets • AAV – current Block 1.5.2 • Verification will be documented in AIPs and AIRs • Verification has 3 parts • Build-2-Build tests (continue the baseline Pedigree from Block 1.0, SVRs, MX builds) • Exclusion/Fill Requirements verified separately • Quality Flags • OPS data sets plus minimal IDPS-altered non-nom granules (from OPS granules)
Block 1.5.2 IDPS PRO EDRPR-related (AAV) Requirements • For each SDR/EDR we have a set of 3 requirements • A(Basic EDRPR functionalities): The Processing SI shall generate the xxx xDR as specified in section x.y of the JPSS Environmental Data Record (EDR) Production Report for NPP, 474-00012 • B (EDRPR Fill exclusions) : The Processing SI shall provide fill values for the xxx xDR in accordance with section x.y of the JPSS Environmental Data Record (EDR) Production Report for NPP, 474-00012 • (cloudiness exclusions , sza exclusions, etc.) • C (EDRPR Quality Flags): The Processing SI shall generate the xxx xDR Quality Flags that are listed in Table 8-x of the JPSS Environmental Data Record (EDR) Production Report for NPP, 474-00012.
Approach for Verification of “A” (Basic Functionality) Reqts • Reqts with basic functionality per the EDR PR are verified by the pedigree of the B2B Quality testing • Unbroken pedigree on both the Development and Sustainment sides • Sustainment and Development B2B continuous testing • Additional testing for specific items as needed • Tests include • Specific Build-2-Build Statistical Comparisons (IDPS2IDPS) • Specific ”Calculated Truth” Comparisons (IDPS2IDPSTRUTH) • B2B Evaluation attempts to be as quantitative as practical • All comparisons, rationales documented • All artifacts CM’d • OAA B2B testing uses a standard set of IDPS Test data Granules • Transitioning from proxy to May 15 Ops set
Approach for Verification of “B” (Fill Exclusion) Reqts • Fill requirements for exclusions and missing data (cloud exclusions, sza excl, etc) that are in the EDR PR. • Includes Fill Values/Types from EDR PR as well as CDFCB Definitions of Fill Values (MISS, NA, ERR, and DNE fill cases) • Will run “calculated truth” tests that fill values and associated Quality Flags are correct. • Most of these are done in build-to-build IDPS regression tests if granules trigger these behaviors • Will explicitly be analyzed in final AAV verification (AIR) • Test done in appropriate granules for the product (cloudiness, sza threshold, etc.)
Approach for Verification of “C” (Quality Flag) Reqts • Quality Flag Testing • New QF’s • verified in Path C PCR S-AIRs – formal process, reviewed by NASA • Included in B2B testing from that point forward, • B2B Quality Testing: • IDPS B2B testing analyzes differences, rationale • Verifies most quality Flag's that are triggered in the B2B Test Data Granules • Non-nominal tests for specific Quality Flag's that are not triggered in the B2B Test data Granules • Limited specially-created non-nom datasets - ~ 5 separate datasets instead of the 50+ currently in use. • Limited captured special conditions datasets - (Solar, lunar eclipse, Maneuver, others) • These datasets will not trigger every single Quality Flag, but will trigger many of the most used and most important QFs • A few qf’s are not triggered at all in the nominal Test Data Granules or the NN datasets – IDPS believes that these were sufficiently verified in SVRs and do not need special datasets/runs. Identified in xls. • Detailed xls has all the info on which QF’s are verified by B2B, which require more analysis for AAV, which are not specifically verified in AAV.
QF xls - Summary • QF xls was reviewed/discussed in the July 10 and Aug 10 mtgs • xls is intended to list all of the EDRPR Quality flags and to summarize the category of the planned IDPS verification for each quality flag. • How each of the quality flags was verified in the past FPV event • Whether the quality flags are triggered in the IDPS Test data sets • How IDPS/PRO is planning to verify each quality flag for AAV • The EDRPR quality flags are covered under specific Block 1.5.2 AAV IDPS/PRO requirements (under revision, 1 QF reqt per product) • Analysis Inspection Plan (AIP) is written to outline the proposed verification; the AIPs go through the critique and ECR process • Analysis Inspection Report (AIR) is written with the results, consistent with the approved AIP. The AIR also undergoes a critique and ECR process. • This is the current IDPS Plan – can be discussed, modified. But still have to balance completeness, schedule, and risk
VIIRS QF Verification: VIIRS CDCNN Datasets • Carefully Designed Catastrophic Non-Nominal (CDCNN) Dataset • Careful design to remove specific VIIRS EV, Cal, and Eng/Thermal packets through a single granule to trigger as many fill and qf conditions as possible throughout the chain (xls shows a rough idea of algs/qf’s that will be affected). • Example: • M15-Band EV missing affects SDR, Imagery EDR, IST EDR, LST EDR, SST EDR • I1-Band EV missing affects SDR, Imagery EDR, SIC EDR. Snow EDRs, VI EDR • M5 Cal packet missing affects SDR qf’s, downstream cloud EDR qf’s • CDCNN usage allows efficient way to run, trigger, analyze a lot of non-nominal VIIRS behavior • CDCNN datasets will be designed for 1 tropical and 1 polar granule • Each dataset will likely have ~15 EV packet removals (15 different bands ), ~5 Cal packet removals (5 different bands), ~1 Eng packet removal. Specific scans and detectors will be carefully chosen for every packet removal in a granule dependent upon the actual scenario (ocean, land, ice, snow) and cloud cover in that granule. • Share/Use these datasets Internally and Externally
Objective/Agenda • Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPE at 7/10 and 8/10 meetings) • Algorithm specific info on quality flag triggering • Algorithm specific info on IDPS QF verification plan
CrIS SDR Quality Flags (Mx6.3B and beyond) CrIS RDR Yield MD LEGEND QF QF4, b4 Band/FOV/… SDR Quality MD Imaginary Magnitude Bit Trim Failed QF4, b5 Band/FOV/… Field PCT parameter Invalid RDR QF4, b2 Band/FOV/… Metadata (MD) SDR Quality QF3, b1-2 Band/FOV/… Imaginary Radiance Thresholds Invalid Geolocation QF3, b3 Band/FOV/… Radiance Thresholds Fringe Count Error Detect QF4, b3 Band/FOV/… Invalid Radiometric Calibration MD Fringe Count Error Correction Failed QF3, b8 Band/FOV/… Invalid Spectral Calibration Invalid Radiometric Calibration RDR Impulse Noise QF3, b4-5 Band/FOV/… QF3, b6-7 Band/FOV/… Suspect Neon Calibration QF1, b6 Scan Excess Thermal Drift DS Window Size QF1, b5 Scan Lambda Monitored Quality QF1, b3 Scan Invalid Instrument Temps ICT Window Size QF1, b4 Scan Scan-level Stand-alone QFs Band/FOV/… Stand-alone QFs ICT Temperature Stability ICT Temperature Consistency - Data Gap QF1, b1 - Timing Sequence QF1, b2 - Lambda Monitored QF1, b3 - Lunar IntrustionQF2, b1-2 - Day/Night QF4, b1
Granule level QF’s – General Info • Most granule level summary qf’s are percentages: • Denominator is number of pixels that are not DNE or pixel trim • Numerator is number of pixels (per denom) triggered in a pixel-level qf. • Out of range summary qf’s have numerator and denominator only calculated over non-fill pixels • Exclusions Summary qf’s are per NG-specified specific definitions – (usually) includes all performance and production exclusions (i.e. exclusions where not produced as well as exclusions where produced but under conditions that are not supposed to count towards performance reqts (in original NPOESS Sys Spec).
CrIS SDR GEO QF’s • Missing Ephemeris or Attitude Data • S/C Diary RDR Data (20 sec granules, 1 packet/sec) • CGS also has daily TLE, SGP4 propagation • QF Options • Nominal • Small Gap (<= 30 secs) • Ephemeris and attitude interpolated between ends • Large Gap (greater than small gap) or overlaps Granule Boundary • Ephemeris uses prediction from TLE, interpolation between prediction and S/C diary ephemeris. • Attitude assumed perfect nadir • Missing for Entire granule (Uses Prediction from TLE) • Ephemeris uses prediction from TLE • Attitude assumed perfect nadir
CrIS SDR Info • CrIS SDR granule: • 4 scans • 30 EV FOR per scan • 9 FOV per FOR • 3 bands (LW, MW, SW) • Products are (for each of 3 bands) • Real spectra radiances (4x30x9) x wavenumber (163 SW or 437 MW or 717 LW) • Imaginary spectra radiances (4x30x9) x wavenumber (163 SW or 437 MW or 717 LW) • Noise estimate spectra for radiances (4x30x9) x wavenumber (163 SW or 437 MW or 717 LW) • QF’s in this presentation are those listed specifically in the EDRPR as quality flags. Note that many of these are more like “products” (not flags), and are also included in the CrIS SDR product separate from the QF bytes. (This is ok, just confusing)
CrIS SDR QF’s • Granule-level qf’s • CrIS RDR Yield • CrIS SDR Yield • Invalid Radiometric Calibration Yield (Red was verified in FPV SVR, Blue was verified IPAC)
CrIS SDR QF’s • Scan-level qf’s • ICT Temperature Stability • Lambda monitored field • Data Gap • Timing Sequence Error • Excess Thermal drift • Suspect Neon Cal • Lambda Monitored Quality • Invalid Instrument Temps • ICT Temperature Consistency • Number of Valid PRT Temps • Measured Laser Wavelength • Lambda Resampling (or Resampling Laser wavelength) (Red was verified in FPV SVR, Blue was verified IPAC)
CrIS SDR QF’s • Band/FOV/scan • Lunar Intrusion • Deep Space Symmetry • DS Spectral Stability • ICT Spectral Stability • DS Window size • ICT Window size (Red was verified in FPV SVR, Blue was verified IPAC)
CrIS SDR QF’s • Band/FOV/FOR/scan-level qf’s • Bit trim failed (from instrument on board) • Fringe Count Error detect (from instrument on board) • Invalid RDR data (from instrument on board) • CrIS SDR Quality • Impulse Noise Count (from instrument on board) • Day / Night • Invalid Geolocation • Invalid Radiometric Calibration • Invalid Spectral Cal • SDR Fringe Count • FCE Correct Fail • Invalid Imaginary Radiance (new in 6.3) (Red was verified in FPV SVR, Blue was verified IPAC)
CrIS SDR Granule-level QF’s • CrIS RDR Yield • This qf summarizes the number of good quality Earth Scene (ES) observations in a granule (accounting for each ES observation per band/FOV/FOR/scan/granule). As directed in TM NP-EMD-2007.510.0027.RevB, the numerator is the number of ES observations where Bit Trim Failed (QF4, bit 4), FCE Detect (QF4, bit 3), and Invalid RDR (QF4, bit 2) quality flags are not set. The denominator is the total number of ES observations (excluding those with DNE fill). The number of ES observations per granule = [scans_per_granule x nBands x nFOVs x nFORs] (4x3x9x30 in a non-short granule). • CrIS SDR Yield • This qf summarizes the number of good quality SDR spectra in a granule (accounting for each ES observation per band/FOV/FOR/scan/granule). As directed in TM NP-EMD-2007.510.0027.RevB, the numerator is the number of ES observations where CrIS SDR Quality QF (QF3, bits 1 and 2) is set to good (= 0) or degraded (= 1). The denominator is the total number of ES observations (excluding those with DNE fill). The number of ES observations per granule = [scans_per_granule x nBands x nFOVs x nFORs] (4x3x9x30 in a non-short granule).
CrIS SDR Granule-level QF’s • Invalid Radiometric Calibration Yield • This qf summarizes the quality of the ES radiometric calibration in a granule (accounting for each ES observation per band/FOV/FOR/scan/granule). The numerator is the number of ES observations where the Invalid Radiometric Calibration flag (QF3, bits 4 and 5) is set to invalid. The denominator is the total number of ES observations (excluding those with DNE fill). The number of ES observations per granule = [scans_per_granule x nBands x nFOVs x nFORs] (4x3x9x30 in a non-short granule).
CrIS SDR Scan-level QF’s • ICT Temperature Stability • This SDR scan-level field (32-bit floating point; degrees Kelvin) is output for each of the two Platinum Resistance Temperature (PRT) measurements. As stated in TM NP-EMD-2007.510.0027.RevB, • this flag is calculated using the following equation: • Where ICTWS is the ICT Window Size, , where T(i,k) is the converted PRT temperature in the validated kth epoch, represents the average ICT temperature measured from a PRT for the ith scan. • Lambda Monitored Field • The Lambda Monitored field is the scan-level SDR field MonitoredLaserWavelength [64-bit floating point, in nm “1e-9 meter”]. It represents the monitored laser metrology wavelength, calculated using data from the 4-min engineering packet (APID 1290) and Neon calibrated laser metrology wavelength. While the measured laser wavelength is computed using only the on-board Neon lamp measurement, the monitored laser wavelength also uses the laser diode temperature.
CrIS SDR Scan-level QF’s • Data Gap • The flag is set for the current scan when data is missing from either the preceding or succeeding scan. • Timing Sequence Error • This flag is set if the time difference between a FOR packet and the corresponding FOR packet in the preceding or succeeding scan is not within 8 seconds plus or minus the timingSequenceErrorThreshold (1e5 microseconds in the PCT).
CrIS SDR Scan-level QF’s • Excess Thermal drift • The Excess Thermal Drift QF is a scan-level QF contained in QF1 bit 5. As described in TM NP-EMD-2007.510.0027.RevB, the flag is set when one or more of the monitored instrument temperatures has exceeded a specified threshold value as follows: • Suspect Neon Cal • As described in TM NP-EMD-2007.510.0027.RevB, the Suspect Neon Calibration QF is set if the fraction of SDRs with invalid neon calibrations exceeds the invalidNeonCalibrationPercentageThreshold value (found in the PCT file). • The value in the PCT file is 25% - this value is explicitly listed in the EDRPR qf description. • Lambda Monitored Quality • As described in TM NP-EMD-2007.510.0027.RevB, the flag identifies an invalid laser wavelength calculation due to invalid diode current and/or temperature measurements. The QF is set when the laser wavelength “lambda monitored” drift, due to drift in the laser diode current and/or temperature, exceeds the laserWavelengthDriftTolerance (ppm) configuration parameter in the PCT file.
CrIS SDR Scan-level QF’s • Invalid Instrument Temps • As described in TM NP-EMD-2007.510.0027.RevB, the flag is set when one or more of the monitored instrument temperatures has exceeded a specified threshold value as follows:
CrIS SDR band/FOV/scan-level QF’s • Lunar Intrusion • This flag is set when the Deep Space (DS) views are contaminated by the moon. During each scan, the instrument takes Earth View (EV) measurements followed by DS and Internal Calibration Target measurements. The DS views in the current scan are used in the calibration of the EV interferograms in the next scan. If lunar intrusion is detected in the current scan, the Lunar Intrusion QF will be set for EV interferograms in the next scan and the DS view will not be used in calibration. • Lunar Intrusion is detected by comparing the current DS radiance with a running average from previous scans. If the difference between the current radiance and the running average is greater than a configurable threshold (maxLunarRadiance in PCT file), the Lunar Intrusion QF is set. Lunar Intrusion is checked for both forward and reverse sweep directions.
CrIS SDR Band/FOV/For/scan-level QF’s • Bit Trim Failed • As described in TM NP-EMD-2007.510.0027.RevB, the flag identifies the IGMs that were clipped during the bit trimming process and thus excluded from SDR processing. The flag is set by the CrIS instrument on-board and contained in the CrIS RDR data packets: • INF_<band><FOV><scene>_ScanStatFlg(I Dat Bit Trim Failure) (Start bit 33, Bit Size = 1) • INF_<band><FOV><scene>_ScanStatFlg(Q Dat Bit Trim Failure) (Start bit 34, Bit Size = 1) • Fringe Count Error detect • As described in TM NP-EMD-2007.510.0027.RevB, the flag identifies the situation where fringes are missed, shifting the interferogram (IGM) Zero optical Path Difference (ZPD) and thus excluding the interferogram from SDR processing. At the ZPD, the phase is mainly derived from the IFGM kernel. The flag is set by the CrIS instrument on-board and contained in the CrIS RDR data packets: • INF_<band><FOV><scene>_ScanStatFlg(Fringe Cnt Err-Smple Pulse Err) (Start bit 35, Bit Size = 1)
CrIS SDR Band/FOV/For/scan-level QF’s • Invalid RDR Data • As described in TM NP-EMD-2007.510.0027.RevB, the flag identifies the case when the CrIS instrument suffered from operational errors (e.g., instrument porch swing direction is not right), hence the pertaining IGMs are excluded from SDR processing. The flag is set by the CrIS instrument on-board and contained in the CrIS RDR data packets: • INF_<band><FOV><scene>_ScanStatFlg(Invld Interferogram Dat Exceed ZPD Sat Limit) (Start bit 32, Bit Size = 1) • Impulse Noise Count • This is the number of the IGM samples that exceeded the impulse noise mask (impulseNoiseCountThresh (PCT parameter)). The field is set by the CrIS instrument on-board and contained in the CrIS RDR data packets: • INF_<band><FOV><scene>_ImpulseNoiseCnt(Number Impulses Detected) (Start bit 54, Bit Size = 10) • Day / Night • The Day / Night QF (QF4, bit 1) is set per band/field of view (FOV)/field of regard (FOR)/scan. As described in TM NP-EMD-2007.510.0027.RevB, the Day / Night QF is set to Day if the local solar zenith angle is less than 90 degrees, and to Night when the local solar zenith angle is greater than or equal to 90 degrees.
CrIS SDR Band/FOV/For/scan-level QF’s • CrIS SDR Quality • As described in TM NP-EMD-2007.510.0027.RevB, the flag is set as follows: • 2 (Invalid): • Bit Trim Failed (QF4 bit 4) = 1; OR • FCE Detect (QF4 bit 3) = 1; OR • Invalid RDR Data (QF4 bit 2) = 1; OR • Invalid Radiometric Calibration (QF3 bits 4-5) = 2; OR • Invalid Spectral Calibration (QF3 bits 6-7) = 2; OR • Radiance Value is less than the minimum radiance threshold or greater than the maximum radiance threshold • Imaginary Radiance Flag triggered(MX6.3) • 1 (Degraded): • Invalid Geolocation (QF3 bit 3) = 1; OR • Invalid Spectral Calibration (QF3 bits 6-7) = 1; OR • Invalid Radiometric Calibration (QF3 bits 4-5) = 1 • 0 (Good): None of the above. • The minimum and maximum radiance threshold values are defined in the PRO include file ProSdrCrisGbl.h as follow: • // Define the min/max values for the radiance values • const Int16 LW_MAX_RADIANCE = 300; • const Int16 MW_MAX_RADIANCE = 200; • const Int16 SW_MAX_RADIANCE = 20; • const Int16 LW_MIN_RADIANCE = -300; • const Int16 MW_MIN_RADIANCE = -200; • const Int16 SW_MIN_RADIANCE = -20;
CrIS SDR Band/FOV/For/scan-level QF’s • Invalid Geolocation • The Invalid Geolocation QF is set if: • Exit vector from the instrument does not intersect the ellipsoid, OR • Ephemeris and attitude data does not cover the granule time span (both on-board ephemeris and TLE derived ephemeris are unavailable), OR • Granule times are bad
CrIS SDR Band/FOV/For/scan-level QF’s • Invalid radiometric cal • As described in TM NP-EMD-2007.510.0027.RevB, the QF is set as follows: • 2 (Invalid): • Radiometric calibration is not performed or performed with invalid calibration data (i.e., DS Window Size = 0 or ICT Window Size = 0) • 1 (Degraded): • [DS Window Size (per band/FOV/DS View) < == 14)AND DS Window Size >= 1; OR • [ICT Window Size (per band/FOV/DS View) < = 14 AND ICT Window Size >= 1; OR • Excess Thermal Drift (scan-level QF1 bit 5) = 1; OR • Invalid Instrument Temperatures (scan-level QF1 bit 4) = 1; OR • ICT Temperature Stability (per scan/ICT View) > ictTempStabilityThreshold (PCT parameter); OR • ICT Temperature Consistency (per scan) > ictTempConsistencyThreshold (PCT parameter); OR; • Number of Valid PRT Temperatures (per scan/ICT View) < numOfValidPRTTempThreshold (PCT parameter); OR • RDR Impulse Noise Count (per band/FOV/FOR/Scan) > impulseNoiseCountThresh (PCT parameter) • 0 (Good): None of the above.
CrIS SDR Band/FOV/For/scan-level QF’s • Invalid Spectral Cal • As described in TM NP-EMD-2007.510.0027.RevB, the flag is set as follows: • 2 (Invalid): • FCE Correction Failed (QF3 bit 8) = 1 (i.e., FCE failed correction); OR • [Suspect Neon Calibration (scan-level QF1 bit 6) = 1 AND Lambda Monitored Quality (scan-level QF1 bit 3) = 1] • 1 (Degraded): • [Suspect Neon Calibration (scan-level QF1 bit 6) = 1 AND Lambda Monitored Quality (scan-level QF1 bit 3) = 0]; OR • [Suspect Neon Calibration (scan-level QF1 bit 6) = 0 AND Lambda Monitored Quality (scan-level QF1 bit 3) = 1]; OR • FCE Detection (QF4 bit 3) = 1 • 0 (Good): None of the above
CrIS SDR Band/FOV/For/scan-level QF’s • Invalid Imaginary Radiance • Triggered by exceeding min and max imaginary radiance thresholds in CrIS PCT file • lwImagRadUpperThresh, • lwImagRadLowerThresh, • mwImagRadUpperThresh, • mwImagRadLowerThresh, • swImagRadUpperThresh, and • swImagRadLowerThresh)
Objective/Agenda • Summarize the IDPS Algorithm Analysis Verification (AAV) Plan (approved by DPE and DPE at 7/10 and 8/10 meetings) • Algorithm specific info on quality flag triggering • Algorithm specific info on IDPS QF verification plan
CrIS SDR QF Verification Summary • Ops data set (15 May) has not yet been analyzed for CrIS triggers. • We are hoping to find triggers for: • Invalid Rad Cal • Invalid Spectral Cal • DS Symmetry • Measured Laser Wavelength • Lambda Resampling • We are planning to “make” or find Non-nominal datasets for: • Data Gap • Maneuver (triggers Invalid Geo)
VIIRS SDR QF Verification • Review QF xls: • CrIS SDR and GEO
References • FPV SVRs –Erooms at Ground Data Product Engineering > IDPS SVRs • IDPS AAV Info – Erooms at Ground Data Product Engineering > IDPS AAV Verification