Hidden Error Variances and the optimal combination of static and flow dependent variances

1. Hidden Error Variancesand the optimal combination of static and flow dependent variances Craig H. Bishop Elizabeth A Satterfield Kevin T. Shanley, David Kuhl, Tom Rosmond, Justin McLay and Nancy Baker Naval Research Laboratory Monterey CA November 2, 2012

2. Introduction: Definitions • Error Variance: Mean of a large number of squared forecast errors. • Flow Dependent Error Variance: Mean of a large number of squared forecast errors given a particular flow. (In order to obtain a large number of errors the “flow” or “condition” must repeat itself). • Hidden Error Variance: A flow dependent error variance that is formally unobservable because the particular flow does not repeat itself. “A conundrum of predictability research is that while the prediction of flow dependent error distributions is one of its main foci, chaos hides flow dependent forecast error distributions from empirical observation.” Bishop and Satterfield (2012a,b, MWR, in press), Satterfield and Bishop (2012ab, to be submitted)

3. 7 60 a) 850-hpa T (oC) b) 850-hpa Td (oC) 45o 45o 6 50 5 40 4 30 ET-ctl 3 ET-ens 20 2 10 1 0 0 0 1 2 3 4 5 6 7 0 10 20 30 40 50 60 Binned Ensemble variance Binned Ensemble variance 50 c) 850-hpa u (m s-1) d) 850-hpa v (m s-1) 45o 45o 40 40 30 30 20 20 10 10 0 0 0 10 20 30 40 0 10 20 30 40 50 Binned Ensemble variance Binned Ensemble variance Previous work: spread-skill diagrams Collect innovations (ob – fcst) corresponding to similar ensemble variances into a bin. Compute bin averaged squared innovation. It should increase with ensemble variance. 600 48-h total precipitation (mm) 45o ET-ctl 500 Binned squared Error (mm)^2 400 ET-ens 300 200 Mean ET-ctl = 119.41 100 Mean ET-ens = 100.82 0 0 100 200 300 400 500 600 Binned Ensemble variance (mm^2) Spread-skill plot for COAMPS simulations similar to for 48-h total accumulated precipitation (mm)^2. These diagrams do not reveal the climatological range of true error variances, nor the degree of variation of ensemble variance given a true error variance. Significant spread-skill relationships were found for all variables – including precipitation Spread-skill plot for COAMPS simulations relative to the control (solid line) and Mean (dashed line)

4. Overview • Observations of hidden error variances using replicate systems. • Empirical determination of key pdfs. • Analytic model of statistical relationships of ensemble variances and true error variances. • Use of (innovation, ensemble-variance) pairs to estimate parameters of analytic model. • Estimation of optimal weights for Hybrid. • Comparison of performance of Hybrid DA with weights from brute force tuning and weights from hidden error variance theory. • Other Hybrid results • Conclusions

5. What is the true flow dependent error variance ? (Slartibartfast – Magrathean designer of planets, D. Adams, Hitchhikers …) • Imagine an unimaginably large number of quasi-identical Earths.

6. 25000 Lorenz Model ReplicatesReveal Hidden Error Variance • Using a 10 variable Lorenz ’96 model with additive model error and a 20 member Ensemble Transform Kalman Filter (ETKF) data assimilation scheme, we created 25,000 independent time series of analyses and forecasts, each having the same true state but differing random draws of observation error. • True Error Variances were then obtained for each spatio-temporal point by averaging the squared forecast error for this point across the 25,000 replicates. First demonstration of ETKF accurately predicting true flow dependent error variance in non-linear system. Scatter plot of ETKF ensemble variance from a single replicate system as a function of true error variance. The true error variance is estimated from all 25,000 replicate systems. The linear fit to the points on the scatter plot is governed by the equation .

7. (a) M=8 (b) M=4 Controlled accuracy of ensemble variances by degrading ETKF variances • A primary objective isto show how pdf of true error variances given an imperfect ensemble variance changes as the accuracy of the ensemble variance changes. • To do this, we created degraded ensemble variances by sampling a Gamma distribution with mean equal to the ETKF variance and relative variance determined by an “effective ensemble size” M. Examples of assumed likelihood gamma pdfs of ensemble variances with a mean of unity. Panel (a) is for an effective ensemble size of M=8, or equivalently, a relative variance of 2/7. Panel (b) is for an effective ensemble size of 4, or equivalently, a relative variance of 2/3.

8. Histograms of true error variance given an imperfect ensemble variance The histograms give an empirical estimate of the pdf of true error variances given a constrained range of sample variances for an 8 member ensemble. The ranges are given on each figure; they correspond to the 2nd and 34th bins, respectively, of 35 bins of true error variance. The solid lines give the fit of an inverse-gamma function to the distribution of true error variances in each bin. M=8 M=8 Inverse-gamma distribution is a very good fit to empirically derived histogram of true error variances given an ensemble variance for all ensemble variance categories.

9. Climatological pdf of true error variances M=8 Inverse-gamma distribution gives a reasonable fit to empirically derived prior climatological pdf of true error variances. M=8 Prior climatological distribution of true error variances. Bars show the probability density histogram of forecast error variances. Solid line shows the fit of the pdf (eq 4) to the data. The thick dashed line marks the mean of both the pdf and the data.

10. Empirical estimation of pdf of true error variance given ensemble variance from 25000 trials (a) M=8, empirical (b) M=2, empirical Red lines depict empirical estimate of pdf of true error variance (ordinate axis) given fixed values of ensemble variance (abscissa axis). Thin green and blue lines give the mode and mean of the empirical estimates of the mode and mean of these estimates. Panels (a) and (b) show the empirical estimates for random sample ensembles of sizes M=2 and M=8, respectively. The grey shading gives an inverse-gamma pdf fit to the climatological pdf of true error variances.

11. An analytic model of hidden error variance Assumption 1: The error of the deterministic forecast is a random draw from a Gaussian distribution, whose true variance i2 is a random draw from a priorclimatological inverse gamma pdfof error variances.

12. (a) M=8 (b) M=4 An analytic model of hidden error variance Assumption 2: Ensemble variances are drawn from a likelihood gamma pdfof ensemble variances with mean a(i2 - min2)+s2min stochastic

13. An analytic model of hidden error variance Bayes’ Theorem defines the posteriorinverse gamma pdfof error variances given an imperfect ensemble variance si2 Climatological Prior Distribution Likelihood distribution of s2 given a particular 2 It can be shown that using assumptions 1 and 2 in Bayes’ theorem gives a that is itself an inverse gamma distribution.

14. (a) M=8, empirical (b) M=2, empirical Green lines give mode Blue lines give mean Given an ensemble variance, there are a broad range of possible true error variances. Current DA schemes require a single value. For the minimum error variance estimate, use the posterior mean. For the maximal likelihood estimate, … For QC, … (c) M=8, analytic (d) M=2, analytic

15. Posterior mean error variance is a Hybrid combination of static and ensemble variances Implications for Ensemble DA? Implications for 4DVAR? Flow dependent ensemble variance Static climatological mean error variance As the stochastic variation of ensemble variance about the true variance goes to zero, the weight on the ensemble variance goes to 1. If there is any imperfection in the flow-dependent ensemble variance, the optimal error variance estimate gives weight to the climatological covariance. If there is no variance of the true error variance, the weight on the static variance goes to 1. Purely flow dependent error variance models are sub-optimal

16. Problem

17. Solution: Equations that define hidden parameters from data assimilation output

18. Equations recover hidden parameters “observed” by replicate systems • “Observed” values are obtained from 175 DA cycles of the 25,000 “replicate systems” • Minimum, mean and maximum of the values retrieved from 21 single system independent time series of • with n=2,000,000. • Retrieved hidden parameters, var(s2), s2, a and M are shown in plots (a), (b), (c) and (d), respectively • “Light grey bars: M=2, Dark grey bars: M=8 • The “given” ensemble sizes in (d) are the random sample ensemble sizes used to degrade the quality of the ETKF ensemble variance.

19. Recovery of min(sigma^2) is inaccurate when min(sigma^2) is small These tests pertain to synthetic data generated using the analytical model of hidden error variance The “specified” were setequal to values previously retrieved from Lorenz model experiments with a “given” M=8 and differing values of the model error q. Each retrieval is from 2,000,000 (innovation, ensemble-variance) pairs synthetically generated from specified distributions. Each plot summarizes informationfrom 60 independent retrievals. The values marked as min, mean, max and std are the minimum, mean, maximum and standard-deviation of the values retrieved from 60 completely independent synthetically generated data sets.

20. Variation of optimal weights with model error and ensemble size, M Ensemble variance weight in dark grey. Static variance weight in light grey. q gives model error variance parameter M gives an “effective ensemble size” corresponding to the relative variance of a random normal ensemble of size M. Variation of weights for mean of posterior distribution of true error variances with model error q and given effective ensemble size M. Black bars give the weights for the de-biased flow-dependent ensemble variance while grey bars give the corresponding weights for the static mean of the climatological error variances. The weight on the ensemble variance increases with ensemble size The weight on the static variance increases as model error variance increases

21. Use of recoveries in Hybrid DA As a start, let’s guess that the optimal weights for error variance prediction are “useful” weights for error covariance prediction. Static covariance Flow-dependent ensemble prediction Covariance matrix of unavoidable errors(?) {In the following 3 examples, we assume that =0}

22. Application to Hybrid DA: Lorenz model 1, perturbed observations. • A suboptimal M=32 member ensemble is generated using a perturbed observations update. • A climatological error covariance matrix (Pfclimatology) is formed by collecting forecast errors for 100,000 time steps (using an 100% ensemble based error covariance matrix) • Pfhybrid is computed at each time step and used in the ETKF DA scheme to obtain an analysis, which is cycled. • We compute the “best practice” hybrid and the “standard” hybrid for all alpha values for comparison. Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights. “Best Practice” hybrid: The ensemble based Pf is corrected by a factor of

23. Possible approaches to concerns in application of theory to Hybrid DA • The eq’s include a kurtosis term which is likely to be sensitive to data QC decisions based on the size of innovations. • Fortunately, it may be shown that the weight for the ensemble variances is entirely independent of this term . • The weight for the static term can then be obtained by insisting that the average of Hybrid variance be consistent with innovation variance.

24. NAVDAS-AR-Hybrid ResultsLow resolution results Alpha=0.5 Hybrid weights computed for 6 distinct regions using new theory Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights.

25. NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0 RMS wind error radiosondeverification results. Redmeans Hybrid outperformed non-Hybrid.0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

26. NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0 RMS wind error self-analysis verification results. Redmeans Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

27. NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0 RMS error global radiosondeverification results. Redmeans Hybrid outperformed non-Hybrid.0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

28. NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0 Geopotential Height Anomaly Correlation (verification against self-analysis). Redmeans Hybrid outperformed non-Hybrid.0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

29. Conclusions • A simple theory of the relationships between ensemble variances and true error variances has been developed. • This theory provides a new method for estimating from an archive of (innovation, ensemble-variance) pairs • the prior climatological pdf of true error variances, • the likelihood pdf of ensemble variances given a true error variance, • the posterior pdf of true error variances given an ensemble variance, and • the mean of (c) as a weighted sum of a static and an ensemble variance. • The pdfs 2a and 2c are well approximated by inverse-gamma distributions for the Lorenz 96 system. • Result (2d) provides a theoretical justification for Hybrid error covariance models that linearly combine static and flow-dependent covariances. • The ansatz that the optimal weights (2d) for error variances are useful for error covariances is true for the Lorenz ’96 system with perturbed obs Hybrid DA and a low resolution Hybrid-4DVAR version of the Navy’s operational DA scheme. • Enables Hybrid weights to be defined regionally at a fraction of the cost of weights obtained via trial and error. • QC and ensemble post-processing applications are also possible.