Hydrologic Data Assimilation with the Ensemble Kalman Filter

1. Hydrologic Data Assimilation with the Ensemble Kalman Filter Dennis McLaughlin, Parsons Lab., Civil & Environmental Engineering, MIT Dara Entekhabi, Parsons Lab., Civil & Environmental Engineering, MIT Rolf Reichle, NASA Goddard Space Flight Center • Problem context - Mapping continental-scale soil moisture from satellite passive microwave measurements. Problem is spatially distributed, nonlinear, and has many degrees of freedom O(106). Available models of hydrologic system and measurement process are highly uncertain. • The ensemble Kalman filter • Results from a synthetic experiment (OSSE) • An opportunity for multi-scale estimation?

2. Evapotranspiration Precipitation Runoff Soil moisture Infiltration Solar Radiation Sensible and Latent Heat Fluxes Soil moisture Ground Heat Flux Soil Moisture Soil moisture is important because it controls the partitioning of water and energy fluxes at the land surface. This effects runoff (flooding), vegetation, chemical cycles (e.g. carbon and nitrogen), and climate. Soil moisture varies greatly over time and space. Measurements are sparse and apply only over very small scales.

3. Microwave Measurement of Soil Moisture L-band (1.4 GHz) microwave emissivity is sensitive to soil saturation in upper 5 cm. Brightness temperature decreases for wetter soils. Objective is to map soil moisture in real time by combining microwave meas. and other data with model predictions (data assimilation).

4. Case Study Area Aircraft microwave measurements SGP97 Experiment - Soil Moisture Campaign

5. Typical precipitation events and measurement times 5 km 5 km 5 cm 10 cm Relevant Time and Space Scales Plan View Estimation pixels (small) Microwave pixels (large) Vertical Section Soil layers differ in thickness Note large horizontal-to-vertical scale disparity For problems of continental scale we have ~ 105 est. pixels, 105 meas, 106 states,

6. Essential Model Features Microwave radiobrightness (deg. Kelvin, L-band) Uncertain land-atmosphere boundary fluxes Soil properties and land use Random meas. errors Radiative transfer model (Measurement equations) Land Surface Model (State equations) Canopy moisture, soil moisture and temperature Uncertain initial conditions States: Canopy moisture Soil moisture Soil temperature State equations are derived from mass and energy conservation Soil moisture is governed by a 1D (vertical) nonlinear diffusion eq (PDE). Soil temperature and canopy moisture are linear ODEs.

7. p[y(t)| Zi] Std. Dev. Mode Mean y(t) The Estimation (Data Assimilation) Problem What is the “best estimate” of the state y(t), given the vector Zi= [z1, ..., zi] of all measurements taken through ti? Posterior probability densityp[y(t)| Zi] is the ideal estimate since it contains everything we know about y(t) (given Zi ) In practice, we must settle for partial information about this density • Some options: • Variational Approaches: Derive mode of p[y(t)| Zi] by solving least-squares problem. • Extended Kalman Filtering: Uses Gaussian assumption to approximate mean and covariance of p[y(t)| Zi]. Both have serious limitations …… Is there a more efficient and complete way to characterize p[y(t)| Zi] ?

8. Ensemble filtering p[y(ti)|Zi] Update with new measurement (zi+1) p[y(ti+1)|Zi+1] Update with new measurement ( zi+1 ) y j(ti+1| Zi+1) y j(ti| Zi) y j(ti+1| Zi) Propagate forward in time Propagate forward in time Time p[y(ti+1)|Zi] ti ti+1 Time ti+1 ti Divide filtering problem into two steps – propagation and update. Characterize random states with an ensemble (j = 1, … , J) of random replicates: Evolution of posterior probability density Evolution of random replicates in ensemble Ensemble filtering propagates only replicates (no statistics). But how should update be performed? It is not practical to construct and update complete multivariate probability density.

9. The Ensemble Kalman Filter The updating problem simplifies greatly if we assumep[y(ti+1)| Zi+1] is Gaussian. Then update for each replicate is: K = Kalman gain derived from propagated ensemble sample covariance Cov[y(ti+1)| Zi]. After each replicate is updated it is propagated to next measurement time. No need to update covariance. This is the ensemble Kalman filter (EnKF). • Potential Pitfalls • Appropriateness of the Kalman update for non-Gaussian density functions? • Need to construct, store, and manipulate large covariance matrices (e.g. 5000 X 5000 for our example)

10. Estimation error Data assimilation algorithm Observing System Simulation Experiment (OSSE) Mean land-atmosphere boundary fluxes Random model error “True” soil, canopy moisture and temperature Soil properties and land use Land surface model Radiative transfer model “True” microwave radiobrightness Random meas. error Mean initial conditions “Measured” microwave radiobrightness Random initial condition error Soil properties and land use, mean fluxes and initial conditions, error covariances Estimated microwave radiobrightness and soil moisture OSSE generates synthetic measurements which are then processed by the data assimilation algorithm. These measurements reflect the effect of random model and measurement errors. Performance can be measured in terms of estimation error.

11. Synthetic Experiment (OSSE) based on SGP97 Field Campaign Synthetic experiment uses real soil, landcover, and precipitation data from SGP97 (Oklahoma). Radiobrightness measurements are generated from our land surface and radiative transfer models, with space/time correlated model error (process noise) and measurement error added. SGP97 study area, showing principal inputs to data assimilation algorithm:

12. Area-average top node saturation estimation error Normalized error for open-loop prediction (no microwave meas.) = 1.0 Compare jumps in EnKF estimates at measurement time to variational benchmark (smoothing solution). EnKF error generally increases between measurements. Increasing ensemble size

13. Effect of Ensemble Size on Spatial Error Distribution (Day 182.67) 6 % 12 % 18 % 24 % Scattered errors decrease as number of replicates increases

14. Error vs. Ensemble Size Error decreases faster than J0.5 up to ~500 replicates but then levels off. Does this reflect impact of non-Gaussian density (good covariance estimate is not sufficient)?

15. Actual and Expected Error Filter consistently underestimates rms error, even when input statistics are specified perfectly. Non-Gaussian behavior?

16. Ensemble Error Distribution Errors appear to be more Gaussian at intermediate moisture values and more skewed at high or low values. Uncertainty is small just after a storm, grows with drydown, and decreases again when soil is very dry.

17. Model Error Estimates EnKF provides discontinuous but generally reasonable estimates of model error but sample problem. Compare to smoothed error estimate from variational benchmark. Specified error statistics are perfect.

18. Multi-scale Possibilities Tree at time ti Repl.1 Repl.2 Repl.J Treat replicates as sample data, derive tree model that replaces covariance. Use tree to condition on new meas. 1 2 3 4 5 6 7 8 Pixel No. Distance EnKF is severely limited by need to compute large covariance at update step. Can this covariance be replaced by a smaller tree model which can then be used to condition ensemble replicates? Fine scale replicates at time ti . . .

19. Summary • Ensemble filtering provides an efficient reduced rank method to reduce size of large distributed systems (replicates are comparable to reduced rank basis functions). • Ensemble forecasting/propagation characterizes distribution of system states (e.g. soil moisture) while making relatively few assumptions. Approach accommodates very general descriptions of model error. • Most ensemble filter updates are based on Gaussian assumption. Validity of this assumption is problem-dependent. • Updates can be performed using a classical covariance-based Optimal Interpolation/Kriging procedure. Alternative covariance-free (multi-scale or variational) updates may provide similar results with much less computational effort. • Ensemble filtering is a very flexible, efficient, and easy-to-use data assimilation method that may greatly improve our ability to interpret large amounts of remotely sensed hydrologic data