1 / 41

Antje Weisheimer

Multi-model ensemble predictions on seasonal to decadal timescales. Antje Weisheimer. Introduction. A. Murphy (1993): What is a good forecast? Consistency: correspondence between forecaster‘s best judgement and their forecasts

candid
Download Presentation

Antje Weisheimer

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multi-model ensemble predictions on seasonal to decadal timescales Antje Weisheimer

  2. Introduction A. Murphy (1993): What is a good forecast? • Consistency: correspondence between forecaster‘s best judgement and their forecasts • Quality: correspondence between forecasts and matching observations • Multifaceted nature of forecast evaluation • Measure-orineted and distribution-oriented scores • Value: benefits realised by decision makers through the use of the forecasts

  3. Structure • The multi-model concept • Examples: • DEMETER –multi-model seasonal forecasts • EUROSIP –Operational multi-model seasonal forecasts at ECMWF • ENSEMBLES – multi-model seasonal, interannual and decadal forecasts • Others • Summary and outlook

  4. initial conditions limited accuracy of observations  ensemble forecasting technique • boundary conditions soil moisture, sea ice, aerosols • model error • model structure complex representation of physical processes in models  combination of different skilful, quasi-independent models into a multi-model ensemble • parameterisations unresolved processes  stochastic physical parameterisations, see lecture by Judith Berner • physical parameter values not precisely known (eg., cloud related parameters)  perturbed parameter approach (Murphy et al., 2004; Stainforth et al., 2005) • numerical representation resolution, truncation • unknown unknowns ? Sources of uncertainty in dynamical seasonal forecasting

  5. verification climate system state space model 1 t=T t=0 model 3 model 2 The multi-model ensemble concept Model 2

  6. verification climate system state space t=T t=0 The multi-model ensemble concept multi-model ensemble

  7. t=T t=T t=0 t=0 A C B t=0 t=T The multi-model ensemble concept: Basic scenarios The verification lies beyond all single-model forecasts • multi-model is improved compared to poor models • however, multi-model is worse than good models One single-model ensemble provides the best forecast • compared to this, the multi-model can only be worse • however, compared to all other single-model ensembles, the multi-model is still improved All single-model ensembles lie below / above the veri-fication • Multi-model is impro-ved because of error cancellation Is there ‘the best model’?? Hagedorn et al. (2005)

  8. SST anomalies cumulative distributions model ranking The multi-model ensemble concept: case A case A: error cancellation multi-model verification single-model DEMETER one-months lead SST anomaly hindcasts (left) and cumulative distribution (right) for JJA 1988 at a single grid point in the tropical Pacific. Hagedorn et al. (2005)

  9. SST 1987 rank SST 1988 MSLP 1988 rank rank The multi-model ensemble concept: case B • The identification of ‘the best model’ depends critically on the aspect considered: • variable • region • season • lead time • choice of metric/skill score There is no single best model!

  10. SST anomalies The multi-model ensemble concept: case C • None of the single-model ensembles predicts the anomaly with Prob ≠ 0 •  the multi-model ensemble can never be better than the best single model, but will always be better than the worse single-models • Note: multi-model assigns a higher probability to negative anomalies than most single-model ensembles Hagedorn et al. (2005)

  11. The DEMETER project multi-model of 7 coupled general circulation models • hindcast production period: 1958-2001 • 9 - member IC ensembles for each model • ERA-40 initial conditions • SST and wind perturbations • 4 start dates per year: 1st of Feb, May, Aug, and Nov • 6 month hindcasts http://www.ecmwf.int/research/demeter/

  12. Feb 87 May 87 Aug 87 Nov 87 Feb 88 ... The DEMETER project multi-model of 7 coupled general circulation models 7 models x 9 ensemble members  63-member multi-model ensemble = 1 hindcast Production for 1958-2001 = 44x4 = 176 hindcasts

  13. ECMWF CNRM UKMO MPI ERA40 Nino3 area DEMETER: example of Nino3 SST hindcasts

  14. under- over- dispersive systematic errors rel. spread of the multi-model ensemble vs. climatology DEMETER: capturing the T2m 1989-1998 verification rel. frequency that the verification (ERA-40) lies outside the multi-model ensemble bounding box, based on 6-hourly data Weisheimer et al. (2005)

  15. DEMETER: capturing the T2m 1989-1998 verification Weisheimer et al. (2005)

  16. start date: fraction of grid points (in %) days DEMETER: capturing the T2m 1989-1998 verification 1st month 2nd month 3rd month capture rate over time Weisheimer et al. (2005)

  17. DEMETER: capturing the T2m 1989-1998 verification multi-model ens single-model ens rel. frequency that the verification lies out-side the ensemble bounding box Weisheimer et al. (2005)

  18. SST MSLP multi-model baseline model ranking DEMETER: multi-model vs single-model Relative ACC improvement of the multi-model compared to the single models for JJA from 1980-2001 (one month lead) Anomaly Correlation Coefficients (ACC) Hagedorn et al. (2005)

  19. SST MSLP multi-model baseline model ranking DEMETER: multi-model vs single-model Relative improvement of the multi-model compared to the single models for JJA from 1980-2001 (one month lead) for different scores. Tropics Anomaly Correlation Coefficients (ACC), root mean square skill score (RMSSS), Ranked Probability Skill Score (RPSS) and ROC Skill Score (ROCSS) Hagedorn et al. (2005)

  20. 1959-2001 single-model multi-model DEMETER: Brier score of multi-model vs single-model Brier skill score Hagedorn et al. (2005)

  21. Resolution skill score Reliability skill score single-model single-model multi-model multi-model DEMETER: Brier score of multi-model vs single-model • improved reliability of the multi-model predictions • improved resolution of the multi-model predictions Hagedorn et al. (2005)

  22. 0.039 0.899 0.141 0.095 0.926 0.169 -0.001 0.877 0.123 0.039 0.899 0.140 0.204 0.990 0.213 0.047 0.893 0.153 0.065 0.918 0.147 -0.064 0.838 0.099 multi-model DEMETER: multi-model vs single-model BSS Rel-Sc Res-Sc Reliability diagrams (T2m > 0) 1-month lead, start date May, 1980 - 2001 Hagedorn et al. (2005)

  23. multi-model minus random chosen single model multi-model minus best single model DEMETER: multi-model vs single-model RPSS, precipitation, 1-month lead, start date November

  24. DEMETER: impact of ensemble size • Is the multi-model skill improvement due to • increase in ensemble size? • using different sources of information? • An experiment with the ECMWF coupled model and 54 ensemble members to assess • impact of the ensemble size • impact of the number of models

  25. 0.170 0.959 0.211 0.222 0.994 0.227 Hagedorn et al. (2005) DEMETER: impact of ensemble size 1-month lead, start date May, 1987 - 1999 BSS Rel-Sc Res-Sc Reliability diagrams (T2m > 0) 1-month lead, start date May, 1987 - 1999 single-model [54 members] multi-model [54 members]

  26. realisations of different multi-model combinations realisations of different single-model ensembles with the same number of members DEMETER: impact of number of models Hagedorn et al. (2005)

  27. DEMETER: prediction of tropical storms • GCMs nowadays are able to simulate tropical storms with a seasonal evolution and interannual variability consistent with observations over the western North Atlantic, eastern North Pacific and western North Pacific • frequency of simulated tropical storms is strongly correlated with interannual variability of observed large-scale circulation • operational monthly forecasts of tropical storm frequency at ECMWF (see lecture by Frédéric Vitart) • objective procedure for tropical stormsdetection tracks low vortices with a warm core above (Vitart et al., 2003) • quality of seasonal prediction of tropical storms may be improved by multi-model combination  DEMETER (Vitart, 2006)

  28. MULTI-MODEL DEMETER: averaged number of tropical storms 1987-2001 Vitart (2006)

  29. observations multi-model forecast 2s error r=0.62 Eastern North Pacific North Atlantic r=0.56 South Pacific r=0.62 Western North Pacific r=0.72 DEMETER: tropical storms interannual variability 1987-2001 Vitart (2006)

  30. EUROSIP: European operational Seasonal-to-Interannual Predictions • Three coupled seasonal forecast systems: • ECMWF • Météo France • UK Met Office • All systems are running on ECMWF supercomputers • Hindcast periods • 1987-2001 for ECMWF and UK Met Office • 1993-2004 for Météo France • Development of multi-model products is ongoing

  31. ensemble mean anomalies forecasts started in Nov 2005 EUROSIP: European operational seasonal multi-model predictions observed SST anomalies DJF 2005/2006 ECMWF Météo France UK MetOffice Courtesy L.Ferranti

  32. EUROSIP: The European winter DJF 2005/2006 Probabilistic multi-model forecasts Prob (T2m > upper tercile) Prob (MSLP<lower tercile) observed anomalies T2m MSLP

  33. EUROSIP: The latest forecasts for JJA 2006 EUROSIP forecasts for JJA initialised on April 1st 2006 Chances for a warm and dry summer are…

  34. stream 1 month 18-24 • Three approaches to tackle model uncertainty: • Multi-model: 7 coupled GCMs, each 9 IC ensemble members • Perturbed physics: 2 coupled GCMs, each 9 IC ens. members • Stochastic physics: 1 coupled GCM, 9 ensemble members • - hindcast production period: 1991-2001 • - seasonal runs (7 months): two start dates per year (May, Nov) • - annual runs (14 months): at least one start date per year (Nov) • - multi-annual/decadal runs (10 years): starting in 1965 and 1994 • - model level data available for 3 of the multi-model GCMs ENSEMBLES: seasonal, interannual and decadal predictions EU funded Integrated Project 09/2004 - 08/2009 http://ensembles-eu.metoffice.com/index.html http://www.ecmwf.int/research/EU_projects/ENSEMBLES/index.html public data dissemination

  35. ENSEMBLES: seasonal, interannual and decadal predictions First decadal hindcast experiments using multi-model, stochastic physics and perturbed parameter ensembles SST anomalies in the Nino3 region GloSea (MetOffice) DePreSyS (pert. phys.) ECMWF-CASBS (stoch. phys.) ECMWF obs

  36. 95% 1971-1990 A1B 2081-2100 A2 2081-2100 B1 2081-2100 A1B+A2+B1 Others: IPCC AR4 multi-model climate change simulations multi-model histograms 23 coupled state-of-the-art GCMs run with different emission scenarios central data archive at PCMDI Probability of warm European summers in 2081-2100 Weisheimer and Palmer (2005)

  37. Others: ensemble climate simulations with perturbed parameters Quantifying Uncertainty in Model Predictions (QUMP) using a 53-member ensemble based on perturbed physical parameters climateprediction.net using a multi-thousand member grand ensemble generated by distributed computing temperature distribution pdf of climate sensitivity Murphy et al, 2004 Stainforth et al, 2005

  38. Summary • The quality of seasonal-to-decadal predictions may be improved by using combined forecasts produced by different models (multi-model ensemble forecasts). • Multi-model ensemble forecasting is a pragmatic and efficient method in filtering out model errors present in the individual ensemble forecasts. • Multi-model predictions yield, on average, more accurate predictions than either of the individual single-model ensembles (e.g., DEMETER). • The improvement is mainly due to more consistency and increased reliability.

  39. Outlook A. Murphy (1993): What is a good forecast? • Consistency: correspondence between forecaster‘s best judgement and their forecasts • Quality: correspondence between forecasts and matching observations • Multifaceted nature of forecast evaluation • Measure-orineted and distribution-oriented scores • Value: benefits realised by decision makers through the use of the forecasts Paco’s talk after coffee break!

  40. References (I) • Doblas-Reyes, F.J., M. Déqué and J.-P. Piedelièvre, 2000: Multi-model spread and probabilistic seasonal forecasts in PROVOST. Q.J.R.Meteorol.Soc.,126, 2069-2088. • Doblas-Reyes, F.J., R. Hagedorn and T.N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part II: Calibration and combination. Tellus, 57A, 234-252. • Hagedorn, R., F.J. Doblas-Reyes and T.N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part I: Basic concept. Tellus, 57A, 219-233. • Joliffe, I.T. and D.B. Stephenson (Ed.), 2003: Forecast verification: A practitioner’s guide in atmospheric science. Wiley New York, 240pp. • Murphy, A.H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281-293. • Murphy, J.M. et al, 2004: Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature, 430, 768-772. • Palmer, T.N. et al, 2004: Development of a European multi-model ensemble system for seasonal to inter-annual prediction (DEMETER). Bull. Am. Meteorol. Soc.,85, 853-872. • Stainforth et al, 2005: Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature, 433, 403-406.

  41. References (II) • Weisheimer, A., L.A. Smith and K. Judd, 2005: A new view of seasonal forecast skill: Bounding boxes from the DEMETER ensemble forecasts. Tellus, 57A, 265-279. • Weisheimer, A. and T.N. Palmer, 2005: Changing frequency of occurrence of extreme seasonal temperatures under global warming. Geophys. Res. Lett.,32, L20721, doi:10.1029/2005GL023365. • Vitart, F., D. Anderson and T. Stockdale, 2003: Seasonal forecasting of tropical cyclone landfall over Mozambique. J. Climate, 16, 3932-3945. • Vitart, F., 2006: Seasonal forecasting of tropical storm frequency using a multi-model ensemble. Q.J.R.Meteorol.Soc., 132, 647-666. Special issue in Tellus (2005), Vol. 57A on DEMETER

More Related