Atmospheric data assimilation at NCEP/EMC

1. Atmospheric data assimilation at NCEP/EMC Andrew Collard, Daryl Keist, David Parrish, Ed Safford, Emily Liu, Manuel Pondeca, MiodragRancic, Lidia Cucurull, Haixia Liu, XiuJuan Su, Shun Liu, Wan-Shu Wu, Paul van Delst, Yanqiu Zhu, Mingjing Tong, David Groff, Mike Lueken, George Gayno, Min-Jeong Kim, Yoichiro Ota, Jim Purser, Russ Treadon, and John Derber

2. Outline • Community Development • Ongoing EMC Development • Assimilation techniques • Use of Observations • Maintenance, Collaboration and General Improvements • Future EMC Directions

3. Part of Community Development JCSDA EMC DA Group DTC NESDIS University Research Community GSI AFWA ESRL NCAR GMAO

4. Community development • Major change for EMC DA Team to move into community development • Tool for community development has been subversion and trac • Requires change in how we do work with significant increase in communication among developers • However EMC cannot count on community for critical changes since we have operational responsibility • Also requires changes in partners way of doing work – often more difficult • GSI oversight board • Improve communication among groups • Ensure coding standards are followed • Substantial percentage of the EMC DA Team’s time is used in supporting community. • Most of the GSI updates come from EMC.

5. Assimilation techniques • Hybrid • Implementation of Hybrid/3dvar-EnKF for Global – David Parrish, Russ Treadon, Mike Lueken, Daryl Kleist, Jeff Whittaker (ESRL) • Hybrid for NAM – Wan-shu Wu • Hybrid for HWRF – Mingjing Tong • Dual Resolution for regional – David Parrish • Add 4d capability to hybrid – Daryl Kleist • LETKF in Hybrid – YoichiroOta • 4d-Var – MishaRancic • Constraint • Add surface friction, mixing, precip and cloud physics to strong constraint – Min-Jeong Kim, MishaRancic, Daryl Kleist • Regional Strong Constraint – David Parrish • Cloudy Radiance Assimilation – Min-Jeong Kim, Emily Liu, Yanqiu Zhu, Will McCarty (GMAO)

6. Implementation of Hybrid/3dvar-EnKFDavid Parrish, Russ Treadon, Mike Lueken, Daryl Kleist, Jeff Whittaker (ESRL), Yoichiro Ota (JMA) • Implemented May 22, 2012 • Previous background error covariance (applied operationally) in VAR • Isotropic recursive filters • Poor handle on cross-variable covariance • Minimal flow-dependence added • Implicit flow-dependence through linearization in normal mode constraint (Kleist et al. 2009) • Flow-dependent variances (only for wind, temperature, and pressure) based on background tendencies • Tuned NMC-based estimate (lagged forecast pairs) • Other changes part of hybrid package

7. EnKF/3DVar hybrid J : Penalty (Fit to background + Fit to observations + Constraints) x’ : Analysis increment (xa – xb) ; where xb is a background BVar : Background error covariance H : Observations (forward) operator R : Observation error covariance (Instrument + representativeness) yo’ : Observation innovations Jc : Constraints (physical quantities, balance/noise, etc.) B is typically static and estimated a-priori/offline

8. Hybrid Variational-Ensemble • Incorporate ensemble perturbations directly into variational cost function through extended control variable • Lorenc (2003), Buehner (2005), Wang et. al. (2007), etc. bf & be: weighting coefficients for fixed and ensemble covariance respectively xt: (total increment) sum of increment from fixed/static B (xf) and ensemble B ak: extended control variable; :ensemble perturbation L: correlation matrix [localization on ensemble perturbations]

9. Single Observation Single 850mb Tv observation (1K O-F, 1K error)

10. Single Observation Single 850mb zonal wind observation (3 m/s O-F, 1m/s error) in Hurricane Ike circulation

11. Dual-Res Coupled Hybrid member 1 forecast member 1 analysis EnKF member update member 2 forecast recenter analysis ensemble member 2 analysis member 3 analysis member 3 forecast GSI Hybrid Ens/Var high res forecast high res analysis Previous Cycle Current Update Cycle

12. Retrospective AC

13. Retrospective Die-off curves

14. Retrospective Tropical Winds

16. ___ opr ….. GSI hybrid

17. HVEDAS Extensions and Improvements • Expand hybrid to 4D • Hybrid within ‘traditional 4DVAR’ (with adjoint) • Pure ensemble 4DVAR (non-adjoint) • EnKF improvements • Explore alternatives such as LETKF • Adaptive localization and inflation • Non-GFS applications in development • NAM /Mesoscale Modeling • Hurricanes/HWRF • Storm-scale initialization • Rapid Refresh • Other global models (NASA GEOS-5, NOAA FIM) • NCEP strives to have single DA system to develop, maintain, and run operationally (global, mesoscale, severe weather, hurricanes, etc.) • GSI (including hybrid development) is community code • EnKF used for GFS-based hybrid being expanded for use with other applications (not yet supported by DTC) • Observation impacts and sensitivity, YoichiroOta

18. Observation impacts with various norms Dry total energy norm Kinetic energy norm Average total observation impacts of various observation types. From 12UTC October 21 2010 to 06UTC October 28 2010 (28 cases). Evaluation FT is 24 hours. Impacts are estimated with the serial EnSRF and simple horizontal advection of localization function. Moist total energy norm Serial EnSRF

19. Impact per 1 obs: AIRS moist total energy Serial EnSRFat the same analysis time 292219

20. Assimilation techniques • Hybrid • Implementation of Hybrid/3dvar-EnKF for Global – David Parrish, Russ Treadon, Mike Lueken, Daryl Kleist, Jeff Whittaker (ESRL) • Hybrid for NAM – Wan-shu Wu • Hybrid for HWRF – Mingjing Tong • Dual Resolution for regional – David Parrish • Add 4d capability to hybrid – Daryl Kleist • LETKF in Hybrid and observation sensitivity – YoichiroOta • 4d-Var – MishaRancic • Constraint • Add surface friction, mixing, precip and cloud physics to strong constraint – Min-Jeong Kim, Daryl Kleist • Regional Strong Constraint – David Parrish • GSI Aerosol assimilation – Ho-Chun Huang • Inclusion of NSST capability – Xu Li • Cloudy Radiance Assimilation – Min-Jeong Kim, Emily Liu, Yanqiu Zhu, Will McCarty (GMAO)

21. Cloudy Radiance Assimilation Min-Jeong Kim, Emily Liu, Yanqiu Zhu, Will McCarty (GMAO) • Large numbers of radiances contain cloud (and precipitation) signal. • If cloudy radiances can be properly used, potential for significant improvement in forecasts of temperature, wind, moisture and cloud fields. • Initially addressing simpler problem with microwave and non-precipitating clouds • Planned for initial operational implementation in the next global GSI upgrade (Spring – Fall 2014)

22. Necessary modifications for cloudy radiances • Simulate cloudy radiances (CRTM) • Define control variable(s) for clouds • Define background error for control variable(s) – hybrid? • Define quality control (currently only doing nonscattering clouds) • Define observation errors • Develop forward model for cloud physics • Tangent Linear • Adjoint (for gradient)

23. Read in & distribute Observations, guess fields background error (T ,q, sfc P, u, v ozone) Additional initializations User input & initializations • Outer loop • Calculate simulated obs and sensitivities at obs locations • (radiances and jacobians with CRTM) • Applying bias corrections • Screening observation with QC processes • Set up right hand side of analysis equation • Call inner loop • Compute gradient information • Apply background error • Compute search direction • Compute step size • Update inner loop analysis increment • Update outer loop analysis increment Write analysis & related output GSI w/o cloudy radiances

24. Read in & distribute Observations, guess fields background error (T ,q, sfc P, u, v ozone, clouds) Additional initializations User input & initializations • Outer loop (add analysis control variable) • Calculate simulated obs and sensitivities at obs locations (clear and cloudy radiances) • (radiances and jacobians with CRTM) • Applying bias corrections (revised for cloudy radiances) • Screening observation with QC processes (removing radiances we cannot simulate) • Set up right hand side of analysis equation (including cloud forcing) • Call inner loop • Compute gradient information (through linearization and adjoint of physics) • Apply background error (including cloud control variable) • Compute search direction • Compute step size (including cloudy radiances) • Update inner loop analysis increment (through linearization of physics) • Update Outer Loop analysis increment (through physics update clouds) Write analysis & related output Revised procedure with cloudy radiances

25. Observation operator Community Radiative Transfer Model (CRTM) AMSU-A Observed Tb First-Guess Tb CH 2 31.4 GHz CH 2 31.4 GHz Much warmer than FG CH 15 89.0 GHz CH 15 89.0 GHz Scattering signal in observations. Precipitation+ ice clouds

26. Observation Error (scan angle dependence) Standard Deviation (STD) of First-guess Departure (NOAA-18 AMSU-A) omgbc: O-F after bias correction omgbc after QC: O-F after bias correction and QC passed - Standard deviation depends on scan angle. - Ch1,2,15 and Ch 3 and 4 behave differently.

27. Observation Error Function of Mean Cloud Liquid Water Path (CLWP) Data # < 50 Data # < 50 • FG departure divided by std ratio as a function of scan position were used to get STD in these plots. • Once observation error are defined as function of CLWP, the error value will be rescaled by multiplying with std ratio depending on scan position. Mean CLWP=0.5×(observed CLWP + model CLWP)

28. Quality Control (QC) • 1. Screening cloud affected AMSU-A data • Criteria:(1) scattering index (using ch1, 2, and 15 + ch6 Tb departure) • (2) retrieved clwp + ch4 Tb departure •  New: Screening retrieved averaged CLWP > 0.5kg/m2 over the ocean • 2. Topography effect: for observations at Zsfc > 2km, observation errors get increased. • 3. Transmittance at the top of the model less than 1 : inflating observation error • 4. Too much sensitivity to sfc temperature/sfc emissivity: inflating observation error • 5. New gross check for all-sky(clear and cloudy) radiance data ich: New observation error (i.e. function of averaged CLWP)

29. Moisture Control Variable What differentiates GSI analysis results when different configurations for moisture control variables are used? Cloud water error statistics show “non-Gaussian” distribution. • Approach I • Single control variable: total moisture, qtot(=q+qc)) • UK Met Office has been using qtot as a single moisture control variable • Approach 2 • Keep two separate control variables for water vapor and clouds • However, develop a different form of cloud water related control variable of which error distribution is more Gaussian than qc • ECMWF’s current efforts are toward this approach. • This will be tested for comparisons with Approach 1.

30. Moisture Control Variable What differentiates GSI analysis results when different configurations for moisture control variables are used?

31. Control Exp 1 Exp 3 Exp 2

32. Summary Cloud Assimilation • There has been great progress in assimilating AMSU-A cloudy radiance data in NCEP Global Data Assimilation System (GDAS). • New observation errors and quality control methods, which are applicable for clear and cloudy sky conditions, have been developed. • Testing/comparing different options for moisture control variables are in progress. • Preliminary results from case studies show that cloud fields are now being actively assimilated and cloud analysis fields get much closer to the retrieved values. • Need to use hybrid background error for “cloud balance”. Dual Res??? • Will be used in simplest situations (microwave – non-precipitating) first in operations. (Planned for next Global GSI upgrade)

33. Use of observationsNew data and improved use of current observations • Radar Data • Creation and use of new VAD (after QC) winds – Shun Liu • Ground based Doppler radar data – Shun Liu • Aircraft based Doppler radar data – MingjingTong • Conventional Data • Mesoscalesurface observations – Manuel Pondeca, Steve Levine, XiuJuan Su • Use of visibility and wind speed in RTMA – Yanqiu Zhu • Improve use of significant levels in RAOB profiles – Wan-shuWu • Satellite Data • Use of ATMS and CrIS data - Andrew Collard (ATMS impact experiments run by JCSDA) • Use of consistent physical retrievals for CLW QC – Andrew Collard • Use of AURA MLS ozone profiles – HaixiaLiu • Bias correction enhancements for GSI – Yanqui Zhu • Satellite wind usage upgrade (EUMETSAT and hourly winds) –XiuJuanSu • SSMI/S data – Andrew Collard, Emily Liu, Banghua Yan (NESDIS) and Ellen Liang (NESDIS) • GOES-R/SEVERI radiances – HaixiaLiu • Enhanced assimilation of GPS-RO data – Lidia Cucurull, R.J. Purser

34. Enhanced assimilation of GPS-RO dataLidia Cucurull, R.J. Purser • Implemented along with Hybrid is the use of Bending angle rather than Refractivity • New and improved forward model to reduce eliminate singularities • Data used up to 50km from 30km for refractivities • Improved quality control (little data rejected) • Cucurull, L., J.C. Derber, and R.J. Purser, A bending angle forward operator for GPS Radio Occultation measurements, submitted to JGR.

35. Occulting GPS Ionosphere Neutral atmosphere LEO Earth Radio Occultation concept • An occultation occurs when a GPS (GNSS) satellite rises or sets across the limb wrt to a LEO satellite. • A ray passing through the atmosphere is refracted due to the vertical gradient of refractivity (density). • During an occultation (~ 3min) the ray path slices through the atmosphere Raw measurement: change of the delay (phase) of the signal path between the GPS and LEO during the occultation. (It includes the effect of the atmosphere). GPS transmits at two different frequencies: ~1.6 GHz (L1) and ~1.3 GHz (L2).

36. What do we “want” to assimilate? Raw measurements of phase of the two signals (L1 and L2) s1, s2, Bending angles of L1 and L2 a1, a2 (neutral) bending angle Clocks correction, orbits determination, geometric delay Refractivity a Atmospheric products Ionospheric correction N Abel transfrom T, Pw, P Hydrostatic equilibrium, eq of state, apriori information

37. GPS RO Refractivity vs. Bending angle experiment design • Uses COSMIC 1-6, SAC-C, TerraSAR-X, METOP/GRAS, GRACE-A, C/NOFS • Period 1 February to 22 March 2011 • Non-Hybrid GSI • T382L64

39. Current work • Use of GPS RO data in Super-refraction layers • Usually found in tropics at the top of the boundary layer. Large change in refractive index. • Use of Refractivity data in this case results in negative bias and impacts all data below • Cannot currently measure Bending Angles > .03 - .04 rad because of instrument SNR – but may be able to used data below the Super-refraction layer

40. Non-SR SR

41. Mismatch between top planetary boundary layer Model BA exceeds limit for observation Below, obs rejected by QC

42. Maintenance, Collaboration and General improvements • Improvement of RTMA application – Manuel Pondeca • Improvement of NAM analysis – Wan-shu Wu • Improvement of HWRF analysis – Mingjing Tong • Add global NMMB capability – David Parrish • Update to revised versions of global cloud and precip physics – Min-JeongKim • Radiance maintenance and enhancement • CRTM improvement– Paul VanDelst, David Groff, Quanhua Liu(NESDIS), Yong Chen(NESDIS) • Add climatological CO2, CO, NO2, CH4 -RunhuaYang • Radiance monitoring upgrade – Ed Safford • FOV updates for new satellite instruments – George Gayno, ?? (NESDIS) • General Maintenance and improvements • Maintenance of EMC GSI trunk - Mike Lueken • Code Structure improvement – Mike Lueken, Ricardo Todling (GMAO) • Improve Structure of diagnostic files and unify data monitoring – Andrew Collard, Ed Safford • Code transfer to different machines – Daryl Kleist, Russ Treadon, Others • Code optimization – All • Pre-operational testing and parallels (Russ Treadon, George Gayno and all)

43. Major Future Activities • Cloud Assimilation • Hybrid enhanced (4d+) and extended to other models • New observations – NPP, JPSS, GOES-R, METOP-B, etc. • Aerosol and trace gas assimilation • Improved use of currently used data • Transition between machines • Supporting operations • Improved optimization/code structure • Collaborative research