CrIMSS EDR Algorithm Improvements beyond Provisional Maturity

1. CrIMSS EDR Algorithm Improvements beyond Provisional Maturity Xu Liu* and Susan Kizer*NASA Langley Research CenterMurty Divakarla*, Mike Wilson*, Xiaozhen Xiong*, Changyi Tan*, Eric Maddy#, Antonia Gambacorta@, Nick Nalli+, andFlavio Iturbide*IM Systems Group, Inc. at NOAA/STAR Degui Gu*, Denise Hagan,* and Xia L Ma*, Northrop Grumman Aerospace Systems Chris Barnet& and Mitch Goldberg$&Formerly with NOAA/STAR and currently with STC, Columbia, MD$JPSS/NOAA

2. Outline • Highlights of the CrIMSS EDR algorithm • Path to provisional maturity • Path beyond provisional maturity for improving EDR performance • Fast track improvements • Short-term algorithm improvements • Long term algorithm improvements • Summary conclusions

3. Highlight of the CrIMSS EDR Algorithm Advantages • Build upon heritage cloud clearing method used by AIRS • AIRS: sequential retrieval • CrIMSS: simultaneous retrieval • Physical Optimal Estimation (OE) method • Highly robust minimization scheme (d-rad method) • Simultaneous retrieval to avoid ad-hoc tuning • Inherent retrieval error estimation • Cloud-clearing is within the IR+MW retrieval with good convergence • No regression/NN used (robust under dusty conditions) • Algorithm is designed with dynamic indexing and dimensions • Easy to change variable dimensions (e.g. emissivity retrieval) • Can change retrieval channel selection by simply replacing a new channel-selection LUT w/o recompiling the code • Can be expanded to retrieval more trace gases • Designed to work when no ATMS data is available • NWP profiles will be used

4. Path to Provisional Maturity (Major road map to the EDR testing and validation) • Generation of realistic CrIMSS proxy data (2009) • LaRC provided the CrIS proxy generation Algorithm • MIT provided the ATMS proxy generation Algorithm • Had been used for algorithm tuning/testing/LUT genration • Used for delivering realistic proxy data to different end users • Porting the IDPS OPS code as Raytheon was still in the development stage • LaRC ported the code and NOAA provided the pre-processor • Kept version updates with IDPS code • Easy to modify (forward model runs, testing various proxy data) • Generated some pre-launch LUT ATMS Bias tuning based on 11/11/2011 • NOAA/STAR provided ECMWF matching • Improved MW convergence rate significantly • Analyzed first light CrIS data (2/20/2012) generated by UW • Robust coding • Algorithm tuning and DR submission based on 2/24/12 and 05/15/12 focus data • Updated bias tuning for ATMS and CrIS • Updated climatology covariance matrix (improved MW convegence rate and performance) • Fixed index error in the code (NGAS and LaRC)—improved day time convergence rate • Modified surface air temperature retrieval over land • See Chris’s provisional maturity presentation for more details

5. Path beyond Provisional Maturity (Proposed Changes/Improvements) • Divide the improvements into three categories • Fast track changes: Quick and small changes to code and LUTs • Short-term Changes: Involved code changes and LUTs • Long-term Changes: Code modification to enhance CrIMSS EDR performance and other products

6. Fast-track Summary • Mainly LUT updates • No need to change IDPS code • Can be easily incorporated into the IDPS OPS code • Expect have significant impact on the performance • List of LUT updates • ATMS bias LUT (CrIMSS-MW-BT-BIAS-CORR-LUT) • CrIS bias LUT (CrIMSS-IR-RTM-BIAS-LUT) • Climatology LUT (CrIMSS-CLIM-LUT) • Atmospheric profile LUT • MW surface emissivity LUT • IR surface emissivity LUT • CrIMSS-IR-ATM-NOISE-LUT • CrIMSS-IR-NOISE-LUT • CrIMSS-MW-ATM-NOISE-LUT • CrIMSS-MW-NOISE-AMPL-LUT • CrIMSS-MW-NOISE-LUT

7. Fast-track changes • New ATMS bias LUT (CrIMSS-MW-BT-BIAS-CORR-LUT) • Minimizing cloud contamination in the LUT generation • Use more QC parameters (MW cloud liquid water, ir clear scene ID…) • Study the variance to estimate • Expect to improve consistency between MW-only and MW+IR retrieval • Increased yield and performance from the MW+IR retrievals

8. Fast-track changes • New CrIS bias LUT LUT (CrIMSS-IR-RTM-BIAS-LUT) • Improve clear ocean scene ID, IR surface emissivity calculations.. • Check variable trace gases, scan dependency • De-weigh (or remove) trace channels • Modify climatology LUT… • Trace retrieval (in IDPS code, long term) • Expect to improve near surface retrievals • Improved EDR performance

9. Fast-track changes • New atmospheric and surface climatology covariance LUT • Handle upper atmospheric water better to improve AVMP above 300 mb • Careful look at the stratification • Improve MW emissivity representation • Polarization, ice, snow, more databases • Improve IR emissivity and reflectivity covariance • Adjust the weight of various IR emissivty in the covgenerations • Expect to improve convergence rate and EDR performance

10. Fast-track Changes • New CrIMSS-IR-ATM-NOISE-LUT • Account for trace gases interferences (CFCs, HNO3, N2O, CH4 ) • Account for IR emissivity error near O3 spectral region • Check CrIMSS-IR-NOISE-LUT • Check proper handle of noise due to apodization • Account for pixel difference … • Expect to improve clear/cloudy ID

11. Fast-track Changes • New CrIMSS-MW-ATM-NOISE-LUT • Look into variances of the bias tuning process • Account for errors due surface emissivity • Account for RTM error • Account for error due to Zeeman line splitting • Study the impact of using ATMS TDR (instead of SDR) • New CrIMSS-MW-NOISE-AMPL-LUT • Currently 0.333 for each channel • Use realistic ATMS remapping noise-reduction factor • New CrIMSS-MW-NOISE-LUT • Update to post-lauch noise values • NewCrIMSS-CHAN-SEL-LUT • Study possibility of avoiding trace gas spectral interferences • Trace gas variation are latitudinal/seasonal dependent • A simple bias correction is not enough to remove all the impact

12. Short-term Algorithm Improvements • Changes are minor but require code modifications • Some of DRs already written (see Murty’s slides) • Quick fixes: • Fix cloud noise amplification factor (NAF) coding (very minor) • Fix scene identification logic (minor) • Fix error in altitude calculation during post processing (minor) • Add precipitation algorithm from Ferraro’s group • Constrain for water vapor saturation check between 300 mb to 0.0005 mb (minor) • Fixes take more effort: • Increase the number of hinge point for IR surface emissivity retrieval • Better surface emisssivity first guess for the cloud clearing algorithm • Improve scene classification module to identify clear/overcast scene better

13. Short-term Algorithm Improvements  Initializing altitude levels to zero. Need to start from surface height • Bug in atmospheric altitude calculation • Will impact pressure EDR and final average AVTP and AVMP EDRs outVec.push_back(0.0); for(Int32 j = temp.size()-1; j > 0; j--) { // Calculate the virtual temperatures vt1 = temp.at(j) * (1.0 + (DRY_MOIST_RATIO * moisture.at(j)) / CrIMSS_KILO); vt2 = temp.at(j-1) * (1.0 + (DRY_MOIST_RATIO * moisture.at(j-1)) / CrIMSS_KILO); virTemp = (vt1 + vt2) / 2; gort = gor / virTemp; if (gort > 0.0) { dzp = log(pressLvls.at(j) / pressLvls.at(j-1)) / gort; } alt = outVec.front() + dzp; outVec.insert(outVec.begin(), alt); } }

14. Short-term Algorithm Improvements • Increase the number of hinge points for IR surface emissivity retrieval • Current IR emissivity hinge points cannot represent land emissivity variations • Will degrade land performance • Right now we are relaxing the noise to handle this (not optimal) • Expect to improve ozone retrieval and better EDR performance • Not too much code changes • Effort in updating the climatology covariance matrix and indexes for emissivity and state vectors

15. Short-term Algorithm Improvements • Better surface emisssivity first guess for the cloud clearing algorithm • Having a better first guess will improve the clear CrIS radiance estimate when performing cloud clearing • Read in emissivity map from a static database • LaRC group (Dan Zhou) has generated global emissivity map from IASI hyperspectral data • At least stratify the IR emissivity climatology background to more classes if not using the map • Improve scene classification module to identify clear/overcast scene better • Use CrIS surface channel for clear ID (Geoge Aumann) … • Use MW retrieved cloud amount • Tuning number of cloud formation determination parameters

16. Long-term Algorithm Enhancement • Start with better first guess for CC • Use regression? • Trace gas retrievals • CH4, N2O, CO, and CO2 • Retrieval cloud parameters • Cloud height • Effective cloud fractions • Dealing with high-spectral resolution CrIS data • Channel selection to reduce computation burden • High-res RTM • Better trace gas retrievals

17. Summary • CrIMSS EDR algorithm is build upon AIRS cloud clearing method • Totally different implementation • Simultaneous retrieval • OE for possible error and averaging kernel outputs • Efficient RTM and minimization scheme • Not a lot of code changes needed to achieve provisional maturity performance • Minor bug fixes • Mainly LUT updates • Will follow the same path to improve EDR performance beyond provisional maturity • LUT updates • Minor Code changes • Expect significant performance enhancement • Further enhancement can be achieved • Trace gas retrieval • Cloud information