Brief overview of the Joint Center for Satellite Data Assimilation (JCSDA) &

1. Brief overview of the Joint Center for Satellite Data Assimilation (JCSDA) & Description of an All-Weather One-Dimensional Variational Assimilation/Retrieval System Sid Ahmed Boukabara MSFC/SPoRT Seminar, November 19th 2010

2. The Joint Center for Satellite Data Assimilation (JCSDA) Sid Ahmed Boukabara, Deputy Director, JCSDA and Lars Peter Riishojgaard, Director, JCSDA MSFC/SPoRT Seminar, November 19th 2010

3. NOAA/NESDIS NOAA/NWS NASA/Earth Science Division NOAA/OAR US Navy/Oceanographer and Navigator of the Navy and NRL US Air Force/Director of Weather JCSDA Partners Vision: An interagency partnership working to become a world leader in applying satellite data and research to operational goals in environmental analysis and prediction Mission: …to accelerate and improve the quantitative use of research and operational satellite data in weather, ocean, climate and environmental analysis and prediction models.

4. Agency Executives NASA, NOAA, Department of the Navy, and Department of the Air Force Management Oversight Board NOAA / NWS / NCEP (Uccellini) NASA/GSFC/Earth Sciences Division (Lee, acting) NOAA / NESDIS / STAR (Powell) NOAA / OAR (Atlas) Department of the Air Force / Air Force Director of Weather (Zettlemoyer) Department of the Navy / N84 and NRL (Chang, Curry) JCSDA Executive Team Director (Riishojgaard) Deputy Director (Boukabara) Partner Associate Directors (Lord, Rienecker, Phoebus, Zapotocny) JCSDA Management Structure Advisory Panel Co-chairs: Jim Purdom, Tom Vonder Haar, CSU Science Steering Committee (Chair: Craig Bishop, NRL)

5. New JCSDA short-term goal:(adopted 03/2008) • “Contribute to making the forecast skill of the operational NWP systems of the JCSDA partners internationally competitive by assimilating the largest possible number of satellite observations in the most effective way”

6. JCSDA Science Priorities • Radiative Transfer Modeling (CRTM) • Preparation for assimilation of data from new instruments • Clouds and precipitation • Assimilation of land surface observations • Assimilation of ocean surface observations • Atmospheric composition; chemistry and aerosol Overarching goal: Help the operational services improve the quality of their prediction products via improved and accelerated use of satellite data and related research Driving the activities of the Joint Center since 2001, approved by the Science Steering Committee

7. JCSDA Mode of operation • Directed research • Carried out mainly by the partners • Mixture of new and leveraged funding • JCSDA plays coordinating role • Also accessible to external community (CIs) • External research • Historically implemented as a NOAA-administered FFO, open to the broader research community • Typically ~$1.5 M/year available => revolving portfolio of ~15 three-year projects • Extended to include contracts (administred by NASA) • Visiting Scientists • Open to all experts (global reach) • Main conditions: Have a host at one of the partners and work on a JCSDA-related activity • Results and progress from both directed and external work reported at annual JCSDA Science Workshop (most recent held on May 2010)

8. JCSDA Working Groups • Composed of working level scientists from (in principle) all JCSDA partners, plus additional members where appropriate • Tasked with sharing information and coordinating work where possible • Six WGs formed • CRTM • IR sounders • Microwave sensors • Ocean data assimilation • Atmospheric composition • Land data assimilation

9. Some of JCSDA Past Accomplishments • Common assimilation infrastructure (between NCEP/EMC, NASA/GMAO) • Community radiative transfer model • Common NOAA/NASA/AFWA land data assimilation system • Interfaces between JCSDA models and external researchers • Snow/sea ice emissivity model • MODIS polar winds • AIRS radiances assimilated • COSMIC data assimilation • Improved physically based SST analysis • Advanced satellite data systems such as DMSP (SSMIS), CHAMP GPS, WindSat tested for implementation • Data denial experiments completed for major data base components in support of system optimization (performed @ NASA/GSFC/GMAO)

10. IASI Impact on Standard Verification Scores 1-31 August 2007 NH 500 hPa Height Anom. Cor. SH 500 hPa Height Anom. Cor. IASI Control J. Jung

11. ASCAT Impact Experiments with GFS

12. COSMIC: recent impact • AC scores (the higher the better) as a function of the forecast day for the 500 mb gph in Southern Hemisphere • 40-day experiments: • expx (NO COSMIC) • cnt (operations - with COSMIC) • exp (updated RO assimilation code - with COSMIC) • Many more observations • Reduction of high and low level tropical winds error L. Cucurull

13. Challenges • US falling behind internationally in terms of NWP skill • Risk of falling further behind if no remedies and current readiness for upcoming missions is not improved

14. NOAA/NCEP vs. ECMWF skill over 20+ years

15. Potential Remedies • Bring resources to adequate levels (Human & IT) • Bring science up to standards (4DVAR, etc) • Better leveraging/coordination between partners • Get help from experts (Technology transfer) or better R2O

16. Potential Strategy for R2O Improvement (underway) • Tools to be (1) developed, (2) improved, (3) validated, (4) made portable and (5) modularized or (6) simply made available: • CRTM • GSI • Calibration tools, BUFR tools, • OSSE/OSE • Diagnostic Tools • Etc NOAA Cooperative Institutes NASA Research Institutions In general (Supported by grants, contracts, etc) JCSDA IT Infrastructure Navy AFWA Operational Centers (NCEP,FNMOC, etc) All benefit from improvements being made in Central Testbed

17. Summary • JCSDA Recent refocus on NWP skill to address issue of underperforming US forecast skill • Multi-level efforts needed and underway: • Operational readiness for GOES-R, NPP/JPSS and other missions • Science improvements in Data Assimilation • Set Up of an IT infrastructure (O2R, OSSE/OSE, etc) • Coordination of efforts between JCSDA partners • Potential coordination with other programs? (GOES-R, SpoRT, HFIP, OSD/PSDI, Testbeds, etc) for a better leveraging of efforts/resources? • Continued need for interaction with outside research community

18. Description of an All-Weather One-Dimensional Variational Assimilation/Retrieval System MiRS: A Physical Algorithm for Rain, Cloud, Ice, Atmospheric Sounding, and Surface Parameters Sid-Ahmed Boukabara, Kevin Garrett, Wanchun Chen, Flavio Iturbide-Sanchez, Chris Grassotti and Cezar Kongoli NOAA/NESDIS Camp Springs, Maryland, USA MSFC/SPoRT Seminar, November 19th 2010

19. All-Weather and All-Surface Applicability (or Cloudy/Rainy data assimilation & Surface Handling) General Overview and Mathematical Basis 2 1 3 4 Performance Assessment Summary & Conclusion Contents

20. Retrieval Mathematical Basis Bayes Theorem (of Joint probabilities) In plain words: Main Goal in ANY Retrieval System is to find a vector X with a maximum probability of being the source responsible for the measurements vector Ym Mathematically: Main Goal in ANY Retrieval System is to find a vector X: P(X|Ym) is Max =1

21. Probability PDF Assumed Gaussian around Background Y(X) with a Covariance E Probability PDF Assumed Gaussian around Background X0 with a Covariance B Mathematically: T T T T ì ì ü ü é é ù ù é é ù ù 1 1 1 1 1 1 m m 1 1 m m - - - - æ ö æ ö exp exp æ X X X X ö B B æ X X X X ö exp exp Y Y Y(X) Y(X) E E Y Y Y(X) Y(X) ï ï ï ï - - - - ´ ´ ´ ´ - - ´ ´ - - - - ´ ´ ´ ´ - - æ æ ö ö æ æ ö ö ê ê ú ú ê ú ê ú ç ç ÷ ÷ ç ç ÷ ÷ í í ç ç ÷ ÷ ç ç ÷ ÷ ý ý 0 0 0 0 2 2 2 2 ê ú ê ú ê ê ú ú è è ø ø è è ø ø è è ø ø è è ø ø ï ï ï ï ê ê ú ú ê ú ê ú ë ë û û ë û ë û î î þ þ Core Retrieval Mathematical Basis Maximizing In plain words: Main Goal in ANY Retrieval System is to find a vector X with a maximum probability of being the source responsible for the measurements vector Ym Is Equivalent to Minimizing Mathematically: Main Goal in ANY Retrieval System is to find a vector X: P(X|Ym) is Max Which amounts to Minimizing J(X) –also called COST FUNCTION – Same cost Function used in 1DVAR Data Assimilation System Problem reduces to how to maximize:

22. Jacobians & Radiance Simulation from Forward Operator: CRTM Cost Function Minimization • Cost Function to Minimize: • To find the optimal solution, solve for: • Assuming Linearity • This leads to iterative solution: More efficient (1 inversion) Preferred when nChan << nParams (MW)

23. Assumptions Made in Solution Derivation • The PDF of X is assumed Gaussian • Operator Y able to simulate measurements-like radiances • Errors of the model and the instrumental noise combined are assumed • (1) non-biased and • (2) Normally distributed. • Forward model assumed locally linear at each iteration.

24. Retrieval in Reduced Space (EOF Decomposition) • All retrieval is done in EOF space, which allows: • Retrieval of profiles (T,Q, RR, etc): using a limited number of EOFs • More stable inversion: smaller matrix but also quasi-diagonal • Time saving: smaller matrix to invert • Mathematical Basis: • EOF decomposition (or Eigenvalue Decomposition) • By projecting back and forth Cov Matrx, Jacobians and X Covariance matrix (geophysical space) Diagonal Matrix (used in reduced space retrieval) Transf. Matrx (computed offline)

25. CRTM as the Forward Model • Have a fully-validated, externally maintained forward operator, • Unrivaled leverage (~4 FT working on CRTM at JCSDA plus a number of on-going funded projects with academia, industry to upgrade CRTM ) . Funded by JCSDA • Have access to a model capable of producing not only radiances but also Jacobians • Long-term benefit: stay up to science art by benefiting from advances in CRTM modeling capabilities

26. Radiances Vertical Integration and Post-Processing Rapid Algorithms (Regression) Advanced Retrieval (1DVAR) Vertical Integration & Post-processing TPW RWP IWP CLW 1st Guess Vertical Integration Temp. Profile 1DVAR Outputs Humidity Profile -Sea Ice Concentration -Snow Water Equivalent -Snow Pack Properties -Land Moisture/Wetness -Rain Rate -Snow Fall Rate -Wind Speed/Vector -Cloud Top -Cloud Thickness -Cloud phase Liq. Amount Prof selection Post Processing (Algorithms) Ice. Amount Prof MIRS Products Rain Amount Prof Emissivity Spectrum Core Products Skin Temperature MiRS General Overview

27. Comparison: Fit Within Noise Level ? Yes Simulated Radiances Measurement & RTM Uncertainty Matrix E Update State Vector Forward Operator (CRTM) New State Vector Geophysical Mean Background Geophysical Covariance Matrix B 1D-Variational Retrieval/Assimilation Measured Radiances MiRS Algorithm Solution Reached No Jacobians Initial State Vector Climatology (Retrieval Mode) Forecast Field (1D-Assimilation Mode)

28. Necessary Condition (but not sufficient) F(X) Fits Ym within Noise levels If F(X) Does not Fit Ym within Noise X is not the solution X is a solution X is the solution All parameters are retrieved simultaneously to fit all radiances together Suggests it is not recommended to use independent algorithms for different parameters, since they don’t guarantee the fit to the radiances Parameters are Retrieved Simultaneously If X is the set of parameters that impact the radiances Ym, and F the Fwd Operator

29. Solution-Reaching: Convergence • Convergence is reached everywhere: all surfaces, all weather conditions including precipitating, icy conditions • A radiometric solution (whole state vector) is found even when precip/ice present. With CRTM physical constraints. Current version Previous version (non convergence when precip/ice present)

30.  DMSP SSMIS F16/F18 AQUA AMSR-E   TRMM/GPM/ M-T TMI, GMI proxy, SAPHIR/MADRAS NPP/JPSS ATMS Current & Planned Capabilities  • MiRS is applied to a number of microwave sensors, • each time gaining robustness and improving validation • for Future New Sensors • The exact same executable, forward operator, • covariance matrix used for all sensors • Modular design • Cumulative validation and consolidation of MiRS  POESN18/N19 Metop-A  : Applied Operationally : Applied occasionally : Tested in Simulation

31. All-Weather and All-Surface Applicability (or Cloudy/Rainy data assimilation & Variational Handling of Surface) General Overview and Mathematical Basis 2 1 3 4 Performance Assessment Summary & Conclusion Contents

32. All-Weather and All-Surfaces sensor • Sounding Retrieval: • Temperature • Moisture • Major Parameters for RT: • Sensing Frequency • Absorption and scattering properties of material • Geometry of material/wavelength interaction • Vertical Distribution • Temperature of absorbing layers • Pressure at which wavelength/absorber interaction occurs • Amount of absorbent(s) • Shape, diameter, phase, mixture of scatterers. • Instead of guessing and then removing the impact of cloud and rain and ice on TBs (very hard), MiRS approach is to account for cloud, rain and ice within its state vector. • It is highly non-linear way of using cloud/rain/ice-impacted radiances. • To account for cloud, rain, ice, we add the following in the state vector: • Cloud (non-precipitating) • Liquid Precipitation • Frozen precipitation Cloud-originating Radiance • To handle surface-sensitive channels, we add the following in the state vector: • Skin temperature • Surface emissivity (proxy parameter for all surface parameters) Upwelling Radiance Surface-reflected Radiance Absorption Downwelling Radiance Scattering Effect Surface-originating Radiance Scattering Effect Surface

33. All-Weather and All-Surface Applicability (or Cloudy/Rainy data assimilation & Variational Handling of Surface) General Overview and Mathematical Basis 2 1 3 4 Performance Assessment Summary & Conclusion Contents

34. Vertical Integration and Post-Processing TPW RWP IWP CLW Vertical Integration Temp. Profile 1DVAR Outputs Humidity Profile -Sea Ice Concentration -Snow Water Equivalent -Snow Pack Properties -Land Moisture/Wetness -Rain Rate -Snow Fall Rate -Wind Speed/Vector -Cloud Top -Cloud Thickness -Cloud phase Liq. Amount Prof Post Processing (Algorithms) Ice. Amount Prof Rain Amount Prof Emissivity Spectrum Core Products Skin Temperature MiRS List of Products Official Products Products being investigated • Temperature profile • Moisture profile • TPW (global coverage) • Land Surface Temperature • Emissivity Spectrum • Surface Type (sea, land, snow, sea-ice) • Snow Water Equivalent (SWE) • Snow Cover Extent (SCE) • Sea Ice Concentration (SIC) • Cloud Liquid Water (CLW) • Ice Water Path (IWP) • Rain Water Path (RWP) • Cloud Profile • Rain Profile • Atmospheric Ice Profile • Snow Temperature (skin) • Sea Surface Temperature • Effective Snow grain size • Multi-Year (MY) Type SIC • First-Year (FY) Type SIC • Wind Speed • Soil Wetness Index The following section about performance assessment is a snapshot.

35. Temperature Profile Assessment(against ECMWF) Angle dependence taken care of very well, without any limb correction MIRS ECMWF Note: Retrieval is done over all surface backgrounds but also in all weather conditions (clear, cloudy, rainy, ice) MIRS – ECMWF Diff MIRS – ECMWF Diff N18

36. Moisture Profile(against ECMWF) • Validation of WV done by comparing to: • GDAS • ECMWF • RAOB MIRS ECMWF • Assessment includes: • Angle dependence • Statistics profiles • Difference maps land Bias Sea Stdev When assessing, keep in mind all ground truths (wrt GDAS, ECMWF, RAOB) N18

37. MiRS TPW Retrieval (zoom over CONUS) TPW Global Coverage MiRS GDAS Very similar features to GDAS Smooth transition over coasts

38. Significantly Reduced False Alarms at the Sea-Ice Edges Significantly Reduced False Alarms at the Sea-Ice Edges Significantly Reduced False Alarms at the Sea-Ice Edges Significantly Reduced False Alarms at the Sea-Ice Edges Significantly Reduced False Alarms at the Sea-Ice Edges Significantly Reduced False Alarms at the Sea-Ice Edges RainFall Rate Assessment MiRS Monthly composite (Metop-A) 1DVAR MSPPS Monthly composite (Metop-A) Heritage algorithm: based on physical regression Significant reduction in Rain false alarm using MiRS, at surface transitions and edges

39. No discontinuity at coasts (MiRS applies to both land and ocean) Image taken from IPWG web site: credit to Daniel Villa MiRS RR part of IPWG Intercomparison(N. America, S. America and Australia sites) This is an independent assessment where comparisons of MiRS RR composites are made against radar and gauges data. Image taken from IPWG web site: credit to John Janowiak

40. Independent Validation (IPWG) 2/2 Caution: algorithms perfs depend on how many sensors are used • Monitor a running time series of statistics relative to rain gauges • Intercomparison with other PE algorithms and radar

41. Global Variationally-based Inversion of Emissivity: Routine Assessment MiRS inverts emissivities for all channels, including high-frequency (Inversion performed in EOF space) Emissivity is assessed by comparing it to analytically-inverted emissivity

42. Surface Emissivity Inter-Comparison12/01/2007 – 02/28/2009 Intercomparison between MiRS variational emissivities and analytical ones Desert __ Ocean __ Sea Ice (Antartic) ___ Amazon __ Sea Ice (Arctic) ___ Wet Land __ Es Sea Ice (First Year) ___ Snow __ MiRS N18 MiRS N18 minus GDAS Emissivity difference (MiRS-Analyt) Es GDAS Frequency (GHz) Frequency (GHz) Differences within 2%. Larger diffs noticed for snow (~8%) & Arctic sea-ice (3%). Questions: Tskin used in analytical emiss from GDAS accurate enough? Is assumption of specularity valid for snow and sea-ice?

43. Illustration of High Variability of Emissivity Case area after rain event MiRS N18 retrieved emissivity at 31 GHz ascending node for 2010-10-19, 2010-10-20, 2010-10-22 and 2010-10-23 (from left to right) 37.0 V channel Es 19.35V channel Day in October CPC real-time 24-hour precipitation from 12Z 2010-10-19, 2010-10-20, 2010-10-22 and 2010-10-23 (from left to right) CPC Figures courtesy http://www.cpc.necp.noaa.gov

44. MIRS Emissivity Response to Surface Moisture Variations –Case study- 05/05/07 05/06/07 05/07/07 MSPPS MSPPS MSPPS NEXRAD NEXRAD NEXRAD . • A significant storm system recorded for its wide-spread damage in human life and property • These storms hit the Midwest during May 5-7, 2007, as seen from MSPPS (top) and NEXRAD Radar (bottom) images

45. MIRS Emissivity Response to Surface Moisture Variations –MIRS Emissivity response May 4, 2007 (before the event) Emisivity at 23 GHz Emisivity at 89 GHz Emisivity Spectra (20-160 GHz) May 8, 2007 (1 day after the event, no rain anymore) May 10, 2007 (3 days after event, emiss back to previous state) • MIRS responds to surface wetness variations before (May 4), right after the storm (May 8) and later (May 10). Note the emissivity depression at 21 GHz and the inverted emissivity spectra on May 8, 2007. • Physically-consistent behavior noticed in the emissivity variation

46. MiRS/N18 Sea-Ice Concentration AssessmentComparison with AMSR-E AMSR-E MiRS/N18 All MiRS surface parameters are obtained from emissivity, not TBs (so the validation of these products is an indirect validation of emissivity itself)

47. MiRS/F16 SSMIS Snow Cover Extent (SCE)Comparison with IMS & AMSR-E 2008-11-18 F16 MIRS F16 NRL All MiRS surface parameters are obtained from emissivity, not TBs (so the validation of these products is an indirect validation of emissivity itself) Less Extensive snow cover Extensive snow cover AMSRE IMS False alarms

48. All-Weather and All-Surface Applicability (or Cloudy/Rainy data assimilation & Variational Handling of Surface) General Overview and Mathematical Basis 2 1 3 4 Performance Assessment Summary & Conclusion Contents

49. Summary of Added Values • All physical approach & simultaneous retrieval • Consistent solution that fit the measurements (satisfying a necessary but often overlooked requirement!). • Applicability to all microwave sensors with same code • All-Weather Sounding • Temperature/Moisture sounding in rainy/cloudy conditions using an all-weather RT/Jacobians operator • Emissivity-Based Retrieval of surface paremeters • Higher accuracy of surface products by using Emissivity instead of Radiances (for Wind speed, Soil moisture, Snow, Ice concentration, etc) • Extended retrieval of TPW to land, sea-ice, snow, coasts, sea • Physical Retrieval of atmospheric rain, ice over ocean & land • System is a retrieval & assimilation system

50. Foreseen Scientific Advances • Extension to new sensors: sounders/imagers (ATMS, GPM/GMI, Megha-Tropiques, etc) • Multi-Sensors Synergy • Take advantage of wider spectral coverage to fully characterize surface emissivity and therefore improve surface classification as well as retrieval of other parameters • Take advantage of multi-angle viewing geometries to more accurately sound temperature and moisture • Extension to other spectral Regions (IR). • Feasible since CRTM is valid in all spectral regions • Cloud/Rain/Ice Sounding • Retrieval of cloud and rain in profile form. Combination of sensors, could reduce ill-posed nature of the problem. Many by-products could result from the cloud profiling (cloud top, thickness, bottom, multi-layer nature, mixed phase information, etc) • Better geophysical background characterization