Acknowledgements: - NOAA NESDIS/GOES-R - NOAA NESDIS/JCSDA

1. Warn-on-Forecast and High Impact Weather Workshop,Norman, OK, 8-9 Feb 2012Challenges of assimilating all-sky satellite radiances Milija Zupanski, Man Zhang, and Karina ApodacaCooperative Institute for Research in the AtmosphereColorado State UniversityFort Collins, Colorado, U. S. A.[ http://www.cira.colostate.edu/projects/ensemble/ ] Acknowledgements: - NOAA NESDIS/GOES-R - NOAA NESDIS/JCSDA - NSF Collaboration in Mathematical Geosciences

2. Overview • Relevance of all-sky radiance assimilation • Current status • Challenges • Future

3. Satellite observations AMSU-A GOES-11 SNDR • Remote sensing is the major source of observations - radars - satellites • Satellite data are available everywhere - open ocean - polar regions - other isolated areas on the globe Need to maximize the utility of cloud observations - hurricanes - severe weather

4. Current status of all-sky radiance data assimilation • Most operational centers assimilate only clear-sky radiances • - The wealth of cloud-related measurements is discarded • - However, most high impact weather events are characterized by the presence of clouds and precipitation • - The consequence is a sub-optimal use of satellite observations • Limited research and operational efforts • - Mostly related to the use of variational methods, only recent use of ensemble and hybrid variational-ensemble methods • - All-sky microwave data assimilation operational at ECMWF since 2009 (Bauer et al. 2010; Geer et al. 2010- SSM/I, TMI. AMSR-E) • - Pre-operational testing at NCEP, also pursued at other operational weather centers • Potential benefit of all-sky radiance assimilation is generally accepted, but it is difficult to extract the maximum information from these observations • - modeling of clouds (e.g., microphysics) • - data assimilation methodology • - computing resources, high resolution

5. Impact of cloud clearing (radiance assimilation) Re-development of the TS Erin (2007): Distribution of AMSU-B radiance data in the NCEP operational data stream: (a) all observations, (b) accepted observations after cloud clearing. Data are collected during the period 15-18Z, August 18, 2007. Note that almost all observations in the area of the storm got rejected by cloud clearing. (from Zupanski et al. 2011, J. Hydrometeorology) Need assimilation of all-sky radiances to improve the observation information value

6. Motivation - CIRA DA research • Develop a robust and efficient data assimilation for high impact weather events • - tropical cyclones • - severe weather • Focus on assimilation of cloud and precipitation affected satellite measurements, such as all-sky radiance assimilation • Utilize operational codes as much as possible, focus on realistic issues • - WRF NMM, HWRF • - GSI • - CRTM • Assimilate cloudy radiance from various sources: • - microwave, infrared, lightning • - combine information from different sources to find most beneficial combinations • List of instruments • - New: GOES-R (ABI, GLM), JPSS (ATMS, CrIS), • - Existing: AMSU-A,B, MHS, AMSR-E, TMI, MSG SEVIRI, WWLLN

7. Challenges of all-sky radiance data assimilation • Data assimilation: Methodological and computational issues • Microphysical control variables • - allow cloud observations to impact hydrometeors • Forecast error covariance • - Forecast error covariance needs to be state-dependent, and also to represent dynamical and microphysical correlations • Nonlinearity and non-differentiability of Radiative Transfer (RT) operator • Correlated observation errors • Non-Gaussian errors • Quantifying all-sky radiance information: • - How to provide a maximum utility of these data, and how to measure success? • Other relevant issues: verification, code maintenance, bias correction, … • Everything is connected, need to take into account all components

8. No cloud ice adjustment With cloud ice adjustment 5-10 K > 25 K Temperature analysis increment at 850 hPa Relevance of microphysical control variables Adjustment of microphysical control variables: - provides a more complete control of initial conditions - allows most direct impactof cloud observations on the analysis - critical for high impact weather (e.g., TC and severe weather) Microphysics control variables: impact on DA Physically unrealistic analysis adjustment without hydrometeor control variable (cloud ice in this example)

9. Forecast error covariance Singular value decomposition (SVD) of the forecast error covariance Analysis correction from variationaland ensemble DA can be represented as a linear combination of the forecast error covariance singular vectors ui Structure of forecast error covariance defines the analysis correction! Fundamentally important to have adequate forecast error covariance. For clouds and precipitation, this implies flow-dependent and dynamically meaningful representation of model uncertainties. Use single observation experiment to assess the structure: The quality of data assimilation can be assessed by examining the structure of forecast error covariance!

10. Forecast error covariance: Algebra Complex inter-variable correlations (e.g., standard dynamical variables and microphysical variables) Correlations between microphysical variables Correlations between dynamical variables Cross-correlations between dynamical and microphysical variables • Only Pdd is well known: • Correlations among microphysical variables not well known • Even less known correlations between dynamical and microphysical variables

11. Forecast error covariance: Algebra PM= modeled error covariance Variational methods: Ensemble methods: PRR = reduced rank error covariance Both methods have limitations in representing cloud-related correlations Variational: modeling of cross-covariances, time-dependence Ensemble: reduced rank Hybrid variational-ensemble DA methods are likely the optimal choice for assimilation of cloud-related observations (i.e. all-sky radiances)

12. Single observation of cloud snow at 650 hPa:ensemble DA horizontal response • Corresponds to high-frequency MW radiance observation • WRF model (nest at 3km) • 09 Sep 2012 at 1800 UTC (a) Snow at 650 hPa (Psnow,snow) (b) V-wind at 650 hPa (Pv,snow) A B Horizontal analysis increments for (a) snow, and (b) north-south wind component

13. Single observation of cloud snow at 650 hPa:ensemble DA vertical response (a) Snow at 34N (Psnow,snow) (b) Rain at 34N (Prain,snow) X Vertical analysis increments for (a) snow, and (b) rain. Difficult to model rain-snow correlation: non-centered response and time-dependence

14. Nonlinear observation operators: (Forgotten) Role of Hessian preconditioning • The same cost function can be defined for variational and for Kalman Filter (e.g., EnKF) methods (Jazwinski 1970): • - KF: an explicit minimizing solution of quadratic cost function using Newton method • - VAR: an iterative solution of an arbitrary nonlinear cost function • Nonlinearities increase for precipitation affected radiances - scattering - clouds, aerosol • Hessian preconditioning has to be “balanced” (i.e. similar adjustment in all variables). Otherwise, minimization will create imbalances. Cost function in (a) physical, and (b) preconditioned space

15. Non-differentiable RT observation operators • All-sky radiative transfer calculation has two computational branches: - clear-sky - cloudy and precipitation-affected • Decision about required calculation depends on model variables, thus creates a discontinuity in gradient and/or cost function • Since commonly used iterative minimization is gradient-based, • non-differentiability could have a large impact on the analysis Assimilation of all-sky radiances may benefit from non-differentiable minimization, or other means of addressing discontinuities

16. Wind analysis uncertainty (500 hPa) Degrees of Freedom for Signal (DFS) OBS 89v GHz Tb METEOSAT Imagery valid at 19:12 UTC 18 Jan 2007 Cloud ice analysis uncertainy Degrees of Freedom for Signal (DFS) all-sky radiance observation information content (Degrees of Freedom for Signal – DFS) MW: AMSR-E all-sky radiance data assimilation (Erin, 2007) (from Zupanski et al. 2011, J. Hydrometeorology) IR: Assimilation of synthetic GOES-R ABI (10.35 mm) all-sky radiances (Kyrill, 2007) (from Zupanski et al. 2011, Int. J. Remote Sensing) Analysis uncertainty and DFS are flow-dependent, largest DFS in cloudy areas of the storm.

17. Quantification of Shannon information for all-sky radiances • All-sky radiance observations are correlated • How to measure information from correlated observations? Use mutual information (I) and entropy (H) By definition Use To obtain Mutual information of dependent variables is smaller than mutual information of independent variables Joint entropy H(Y1,Y2) can be used to quantify the loss of information due to correlations between all-sky radiance observations Y1andY2

18. Future • Important to develop capability to extract maximum information from cloudy and precipitation-affected radiances • Critical for improved analysis and prediction of TC and severe weather outbreaks • Take into account all relevant components (e.g., current challenges) • - partial solutions will not bring true progress • - if Gaussian error statistics is not correct, but used, the cost function is inadequate, implying incorrect minimizing solution • - unbalanced Hessian preconditioning will adversely change the adjustment of variables by creating dynamical imbalances of the analysis • Computation • - RT computation increases 2-3 times with scattering • - number of observations increases by an order of magnitude due to cloudy information • - 20-30 times more expensive to compute • Improved information measures for all-sky radiances (e.g., assessment) • Combine information from various sources: GOES-R, JPSS (MW, IR, Lightning)

19. References: Non-differentiable minimization Steward, J. L., I. M. Navon, M. Zupanski, and N. Karmitsa, 2011: Impact of Non-Smooth Observation Operators on Variational and Sequential Data Assimilation for a Limited-Area Shallow-Water Equation Model. Quart. J. Roy. Meteorol. Soc., DOI: 10.1002/qj.935. All-sky IR Zupanski D., M. Zupanski, L. D. Grasso, R. Brummer, I. Jankov, D. Lindsey and M. Sengupta, and M. DeMaria, 2011: Assimilating synthetic GOES-R radiances in cloudy conditions using an ensemble-based method. Int. J. Remote Sensing, 32, 9637-9659. All-sky MW Zupanski, D., S. Q. Zhang, M. Zupanski, A. Y. Hou, and S. H. Cheung, 2011: A prototype WRF-based ensemble data assimilation system for downscaling satellite precipitation observations. J. Hydromet., 12, 118-134.