Huug van den Dool and Steve Lord International Multi Model Ensemble

1. Huug van den Dool and Steve LordInternational Multi Model Ensemble

2. Two Consolidation Projects: • Towards an International MME: CFS+EUROSIP(UKMO,ECMWF,METF) 11 slides • Towards a National MME: CFS and GFDL, and NCAR/ CCM3.0/3.5 and NASA/GFSC 18 slides

3. Does the NCEP CFS add to the skill of the European DEMETER-3 to produce a viable International Multi Model Ensemble (IMME) ? Huug van den Dool Climate Prediction Center, NCEP/NWS/NOAA Suranjana Saha and Åke Johansson Environmental Modeling Center, NCEP/NWS/NOAA August 2007

4. DATA and DEFINITIONS USED • DEMETER-3 (DEM3) = ECMWF + METFR + UKMO • CFS • IMME = DEM3 + CFS • 1981 – 2001 • 4 Initial condition months : Feb, May, Aug and Nov • Leads 1-5 • Monthly means

5. DATA/Definitions USED (cont) • Deterministic : Anomaly Correlation • Probabilistic : Brier Score (BS) and Rank Probability Score (RPS) • Ensemble Mean and PDF • T2m and Prate • Europe and United States “ NO (fancy) consolidation, equal weights, NO Cross-validation”

6. Number of times IMME improves upon DEM-3 :out of 20 cases (4 IC’s x 5 leads): “The bottom line”

7. Frequency of being the best model in 20 casesin terms ofAnomaly Correlation of the Ensemble Mean “Another bottom line”

8. Frequency of being the best model in 20 casesin terms ofRanked Probability Score (RPS) of the PDF “Another bottom line”

9. CONCLUSIONS • Overall, NCEP CFS contributes to the skill of IMME (relative to DEM3) for equal weights. • This is especially so in terms of the probabilistic Brier Score and for Precipitation

10. CONCLUSIONS (Cont) In comparison to ECMWF, METFR and UKMO, the CFS as an individual model does: • well in deterministic scoring (AC) for Prate and • very well in probability scoring (BS) for Prate and T2m over both USA and EUROPEAN domains

11. International Multi-ModelEnsemble (IMME) Status S. Lord, S. Saha, H. Vandendool

12. Status • Goal: produce operational ensemble products from CFS and EUROSIP seasonal climate products • EUROSIP • ECMWF • Met Office • Meteo France • Proposal will be submitted to EUROSIP Council • Covers • Licensing and product distribution • Commercial interest and revenue sharing (none for US) • Consistent with EUROSIP general provisions • Formal Memorandum of Understanding has been drafted • Covers IMME products • Decision expected by end of calendar 2008

13. Status (2) • Some tenets of a potential agreement • NCEP and E-partners will coordinate distribution of IMME products to their users on a regular monthly schedule • Product delivery will not compromise any organization’s operational delivery schedules and commitments • NCEP wishes to join the EUROSIP Steering Group as associate partner (non-voting member) and asks to participate in future meetings • Associated research program possible for product improvement

14. We do work on methods! M. Peña Mendez and H. van den Dool, 2008: Consolidation of Multi-Method Forecasts at CPC. Accepted JCLIM 2008 Unger, D., H. van den Dool, E. O’Lenic and D. Collins, 2008: Ensemble Regression. Accepted MWR Wanqiu Wang: Pdf mapping methods Apply to soil moisture analyses

15. Huug van den Dool, Yun Fan and Malaquias Pena The Multi-Model Ensemble Approach for Soil Moisture Analyses in the Absence of Verification Data .

16. Suppose we want to do MME with EIGHT MODELS 1: R1 2: R2 3: NA(RR) 4: ERA40 5: LB Climate Divisions 6: LB Global 0.5 degree 7: Noah retroactive 8: VIC retroactive Common Period 1979-2001. Monthly mean total column soil moisture data on a 0.5 by 0.5 grid over the US. We know how to take the mean, but how about a weighted mean??

17. Upfront we forgive models for: Error in the mean (most models much too dry in Illinois) Wildly different standard deviations CON is applied to standardized anomalies

18. What is a consolidation (CON)??? K CON = Σαk w k (1) k = 1 i.e. a weighted mean over K model estimates of standardized soil moisture anomalies. One finds the K alphas, the weights, typically by minimizing the distance between CON and observed w for a number of cases.

19. If we had observations for soil moisture we would first do a : Classic or Unconstrained Regression (UR) The general problem of consolidation consists of finding a vector of weights, α, that minimizes the Sum of Square Errors, SSE, given by the following expression: SSE = (Wα - o)T(Wα - o) (2) Then leads to WTWα = WTo So the weights are formally given by α = A-1 b (3) where A = WTW is the covariance matrix, b=WTo and the superscript -1 denotes the inverse operation. Equation (3) is the solution for the ordinary (Unconstrained) linear Regression (UR).

20. Essentially, ridging is a multiple linear regression with an additional penalty term to constrain the size of the squared weights in the minimization of SSE (2): J = (Wα - o)T(Wα - o) + λαTα (4) Minimization of J leads to α = ( A + λ I ) -1b (5) where I is the identity matrix, and , the regularization (or ridging) parameter, indicates the relative weight of the penalty term. Similarities between the ridging and Bayesian approaches for determining the weights have been discussed by Hsiang (1976) and Delsole (2007). In the Bayesian view, (5) represents the posterior mean probability of α, based on a normal a priori parameter distribution with mean zero and variance matrix (σ2/λ)I, where σ2I is the matrix variance of the regression residual, assumed to be normal with a mean zero.

21. Dilemma Outside Illinois we don’t have (sufficient) soil moisture observations to consider CON methods. (Equal weight is always possible of course).

22. Line of Attack In the absence of soil moisture data…we could use co-located Temperature data (two months later) to do a CON (at least in ‘warm’ half of the year). This CON serves CPC’s application. In a sense we weigh models by their ability to predict co-located future temperature (April thru September only). As an aside: We know (and hope) that soil moisture also helps in non-co-located T&P, but we cannot easily work this into a weighting scheme. The local effect on T is undisputed (e.g. dry/wet soil leads to high/low temps – thus expect negative weights!) A hydrologist could do this against runoff obs, an agronomist against crop yields, disease (obs!) over matching years

23. Shown above is the vector bX100 in Eq.(3), which is also the correlation between each model’s soil moisture and the temperature two months later. α = A-1 b (3) • Conclusions • All model’s w correlates negatively with future T. Good! • Some models (the 2 LBs, VIC, Noah ….) correlate a little better (with future T) than others (over 1979-2001) • A skill based weighting scheme without consideration of co-linearity would give the highest weights to these models (CPC ‘standard’) • Correlations (even -0.15) are modest, even if highly significant. Remember: This is an aggregate for all of the US and 6 warm months (April-Sept) combined.

24. Shown are the weights α calculated from Eq. (3), α = A-1 b with minimal ridging. • Conclusions • In the co-linear mix the Leaky Buckets carry most of the weight, followed by Noah and VIC etc. The remaining model speak for portions of the variance that, for the most part, are already accounted for by the leading models. • 2) 75% ridging makes for a stable solution (all weights <=~0.) • Question: • How much better is the weighted average than an equal weight (-1/8th) mean?, and how much better than the best individual model???

25. Conclusions • Equal weight MME is NOT better than the best single model because it gives too much importance to poorly performing models. • Weighted MME is the best!, although the margin of gain may disappoint some of us.

26. RIW RI RIM +CPC skill limit +Delsole equal weight limit Classic Climo UR MMA COR