Data Assimilation Methods Experience from operational meteorological assimilation

1. Data Assimilation MethodsExperience from operational meteorological assimilation John C. Derber Environmental Modeling Center NCEP/NWS/NOAA Workshop on Chemical data assimilation and data needs

2. Data Assimilation Context • Data assimilation attempts to bring together all available information to make the most probable estimate of: • The atmospheric state • The initial conditions to a model which will produce the best forecast. Workshop on Chemical data assimilation and data needs

3. Data Assimilation Context • Information sources • Observations • Background (forecast – carries forward info) • Dynamics (e.g., balances between variables) • Physical constraints (e.g., q > 0) • Statistical relationships between variables • Climatology • Analysis should be most probable state given the above information sources and associated probability density functions Workshop on Chemical data assimilation and data needs

4. Data Assimilation Context • Must build data assimilation system within context of : • Observing system • Data handling system • Forecast model • Verification system • Computational resources • Human resources • Available knowledge about observations and statistics Workshop on Chemical data assimilation and data needs

5. NCEP Production Suite Repeated four times per day Workshop on Chemical data assimilation and data needs

6. Basic Assumptions (often violated) • Data (forecast and most observations) are unbiased • e.g. radiosonde and others commonly biased • All forecast models have significant biases. • Note satellite observations bias corrected. • Observational errors normally distributed • e.g., moisture errors not be normally distributed because moisture cannot be < 0 or >> saturation. • Background uncorrelated to observational errors • May be true if not using retrievals • Representativeness error likely correlated Workshop on Chemical data assimilation and data needs

7. Data assimilation theory • With previous assumptions, most likely solution can be shown to be equivalent to solving variational problem • Variational problem closely related to Kalman filtering/smoothing and ensemble filtering • Often details more important than basic technique Workshop on Chemical data assimilation and data needs

8. Atmospheric analysis problem (theoretical) J= Jb + Jo + Jc J = (x-xb)TBx-1(x-xb) + (K(x)-O)T(E+F)-1(K(x)-O) + JC J = Fit to background + Fit to observations + constraints x = Analysis xb = Background Bx = Background error covariance K = Forward model (nonlinear) O = Observations E+F = R = Instrument error + Representativeness error JC = Constraint term Workshop on Chemical data assimilation and data needs

9. Atmospheric analysis problem (Practical) • Must make analysis problem better conditioned to allow for faster convergence • Current Hessian = B-1 + Kt (E+F)-1K + … where K is the linearization of K • Want something closer to identity matrix • Preconditioning using B matrix • Inclusion of all nonlinearities in minimization not necessary • substantial computation for little change • external iteration • Need to simplify definition of B matrix to make computation feasible. • B matrix series of operators Workshop on Chemical data assimilation and data needs

10. Atmospheric analysis problem (Practical)Analysis variable • Analysis variable (x) and background (xb) • Current global operational length ~ 47,000,000 • (5*nlev+2)*(Ttrunc-1)*(Ttrunc-2) +nchan*npred • 3-D – T382L64 • Temperature spectral coefficients • 2 wind fields of spectral coefficients • Ozone spectral coefficients • Cloud liquid water spectral coefficients • 2-D – T382 • Surface pressure of spectral coefficients • Skin temperature of spectral coefficients • Predictors for bias correction for satellite data • xb background values – 6 hr forecast Workshop on Chemical data assimilation and data needs

11. Sample forecast error structure Workshop on Chemical data assimilation and data needs

12. Sample forecast error structure Workshop on Chemical data assimilation and data needs

13. Sample forecast error structure Workshop on Chemical data assimilation and data needs

14. Necessary future background error development • Current errors are not situation dependent • GSI analysis system under development to include situation dependent errors • Using recursive filters to define background errors • How to define errors still area of active research Workshop on Chemical data assimilation and data needs

15. Workshop on Chemical data assimilation and data needs

16. Workshop on Chemical data assimilation and data needs

17. Atmospheric analysis problem (Practical) • Inclusion of K operator most important advance in meteorological DA • Allows analysis variable to be different than observations • Simplest – 3-D interpolation to obs. location • More complicated – includes radiative transfer calculation • Even more complicated – adds integration of forecast model (4D-Var) Workshop on Chemical data assimilation and data needs

18. Forward radiative transfer models • Use of satellite radiance observations requires development of appropriate radiative transfer model • NCEP using JCSDA Community Radiative Transfer Model (CRTM). • Currently includes sensitivity to temperature, moisture, ozone and aerosols (not completely tested) • Ongoing work to include clouds and other atmospheric constituents Workshop on Chemical data assimilation and data needs

19. Conventional data Radiosondes Pibal winds Aircraft winds and temperatures Ships Buoys Synoptic stations Profilers Satellite data NOAA-14,15,16,17 1b radiances GOES-10,(12) 5x5 radiances Scatterometers SSM/I wind speed and precipitation TRMM precipitation AMWs SBUV ozone profiles Atmospheric analysis problem (Practical)Observations used Workshop on Chemical data assimilation and data needs

20. Atmospheric analysis problem (Practical)Observations used • Over 113M observations received. • Over 7M observations per day used. • Data selection and quality control eliminate many observations • Data selection applied because of: • redundancy in data • reduction in computational cost • eliminate observations which are not useful • E and F assumed diagonal – probably not true (especially for F). Workshop on Chemical data assimilation and data needs

21. More about observations • Observations communicated to NCEP and other NWP centers (GTS +) and available in real time • proprietary observations • Data should be in standard format (BUFR –WMO standard) • Quality control – eliminate bad data and data which cannot be properly modeled • Consistency checks (aircraft track, hydrostatic, etc.) • platform specific checks • cross platform checks Workshop on Chemical data assimilation and data needs

22. Atmospheric analysis problem (Practical)Constraint term • Currently only constraint used is limitation on the relative humidity • moisture > 0. and RH < ~1.2 • Used as penalty term • Applied in inner iteration - nonlinear • Can make problem significantly less well conditioned • May be able to produce similar result through careful design of the analysis variable. Workshop on Chemical data assimilation and data needs

23. Atmospheric analysis problem (Practical)Solution technique • Solution must be made in real time (<20 minutes for global, less for limited area models at NCEP) • At solution = 0 • Use simple conjugate gradient scheme to find minimum • Note one must have adjoint of observation operator to variationally solve problem. Usually not available with new observations. Workshop on Chemical data assimilation and data needs

24. Workshop on Chemical data assimilation and data needs

25. Workshop on Chemical data assimilation and data needs

26. Workshop on Chemical data assimilation and data needs

27. Workshop on Chemical data assimilation and data needs

28. Workshop on Chemical data assimilation and data needs

29. Workshop on Chemical data assimilation and data needs

30. Potential problems for Chem. DA • Sufficient sensitivity to Chem. in observing system • Real-time observation availability and standardized formats and communication procedures • Observation quality control • Development of accurate forward models (and adjoints) and forecast models including Chem. • Defining spatial and inter-variable correlations for Chem. variables • Forecast model, forward model, and observational biases • Developing appropriate validation and data monitoring capabilities • Sufficient computational/human resources • Collaboration Workshop on Chemical data assimilation and data needs

31. Final comments • Assimilation is the integration of all knowledge of the atmosphere (observations, physics, statistics) to estimate the state of the atmosphere • Data assimilation systems are fairly good at large scales for the basic meteorological variables but other variables still in infancy. – Managing expectations • Because of limited resources, operational systems must satisfy multiple purposes and any new developments must fit into operational infrastructure • Must learn to think in terms of increments • In data assimilation details are extremely important – you must do everything well! Workshop on Chemical data assimilation and data needs