Caio A. S. Coelho Department of Meteorology University of Reading c.a.d.s.coelho@reading.ac.uk

1. Forecast calibration and combination: Bayesian assimilation of seasonal climate predictions Thanks to: David B. Stephenson, Magdalena Balmaseda, Francisco J. Doblas-Reyes and Sergio Pezzulli Caio A. S. Coelho Department of Meteorology University of Reading c.a.d.s.coelho@reading.ac.uk • PLAN OF TALK • Calibration and combination issues • Conceptual framework for forecasting • Forecast Assimilation: • Example 1: Nino-3.4 index forecasts • Example 2: Equatorial Pacific SST forecasts • Example 3: S. American rainfall forecasts • Example 4: Regional rainfall downscaling • EUROBRISA project • Met Office, Exeter (U.K.), 20 February 2006

2. This talk is based on the following work: Coelho C.A.S. 2005: “Forecast Calibration and Combination: Bayesian Assimilation of Seasonal Climate Predictions”. PhD Thesis. University of Reading. 178 pp. Coelho C.A.S., D. B. Stephenson, M. Balmaseda, F. J. Doblas-Reyes and G. J. van Oldenborgh, 2005: Towards an integrated seasonal forecasting system for South America. ECMWF Technical Memorandum No. 461, 26pp. Also in press in the J. Climate. Coelho C.A.S., D. B. Stephenson, F. J. Doblas-Reyes, M. Balmaseda, A. Guetter and G. J. van Oldenborgh, 2006: A Bayesian approach for multi-model downscaling: Seasonal forecasting of regional rainfall and river flows in South America. Meteorological Applications, 13, 1-10. Stephenson, D. B., Coelho, C. A. S., Doblas-Reyes, F.J. and Balmaseda, M., 2005: “Forecast Assimilation: A Unified Framework for the Combination of Multi-Model Weather and Climate Predictions.” Tellus A, Vol. 57, 253-264. Coelho C.A.S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 2004: “Forecast Calibration and Combination: A Simple Bayesian Approach for ENSO”. Journal of Climate. Vol. 17, No. 7, 1504-1516. Coelho C.A.S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 2003: “Skill of Coupled Model Seasonal Forecasts: A Bayesian Assessment of ECMWF ENSO Forecasts”. ECMWF Technical Memorandum No. 426, 16pp. Available from: http://www.met.rdg.ac.uk/~swr01cac

3. Calibration and combination issues Calibration • Why do forecasts need it? • Which are the best ways to calibrate? • How to get good probability estimates? • Who should do it? Combination • Why combine forecasts? • Should model predictions be weighted or selected? • How best to combine? • Who should do it?

4. Conceptual framework “Forecast Assimilation” Data Assimilation

5. Multi-model ensemble approach DEMETER Development of a European Multi-Model Ensemble System for Seasonal to Interannual Prediction Model formulation Errors: Initial conditions Multi-model Ensemble Solution: http://www.ecmwf.int/research/demeter

6. DEMETER Multi-model ensemble system 7 coupled global circulation models . 9 member ensembles ERA-40 initial conditions SST and wind perturbations 4 start dates per year (Feb, May, Aug and Nov) 6 month hindcasts . . . Hindcast period: 1980-2001 (1959-2001)

7. Examples of application • • 0-d: Niño-3.4 index • 1-d: Equatorial Pacific SST • 2-d: South American rainfall

8. Example 1: Empirical Niño-3.4 forecasts 95% P.I. Well-calibrated: Most observations in the 95% prediction interval (P.I.)

9. ECMWF coupled model ensemble forecasts m=9 DEMETER: 5-month lead • Observations not within the 95% prediction interval! • Coupled model forecasts need calibration

10. Univariate X and Y Prior: Likelihood: Posterior: Bayes’ theorem:

11. Likelihood modelling: y

12. Combined forecasts  Note: most observations within the 95% prediction interval!

13. Comparison of the forecasts Empirical Coupled • SUMMARY • Combined forecasts: • are better calibrated than coupled • have less spread than empirical • match obs better than either Combined Blue dots = observations Red dots = mean forecast Grey shade = 95% prediction interval

14. Some verification statistics Mean Absolute Error (MAE) defined as: The Brier score (BS) is a simple quadratic score for probability forecasts of binary events (e.g. whether SST anomaly < 0). It is defined as:  Combined forecasts have smallest MAE, BS, and spread

15. Multivariate X and Y: More than one Normal variable Matrices Prior: Likelihood: Posterior:

16. Example 2: Equatorial Pacific SST DEMETER: 7 coupled models; 6-month lead Forecast probabilities: p SST anomalies: Y (°C)

17. Brier Score as a function of longitude Brier Score=0.25 for p=0.5 climatology Brier Score<0.25  more skilful than climatology Forecast assimilation reduces (i.e. improves) the Brier score in the eastern and western equatorial Pacific

18. Brier Score decomposition uncertainty reliability resolution

19. Reliability as a function of longitude Forecast assimilation improves reliability in the western Pacific

20. Resolution as a function of longitude Forecast assimilation improves resolution in the eastern Pacific

21. Why South America?  Seasonal climate potentially predictable DEMETER Multi-model El Niño (DJF) La Niña (DJF) Correlation of ensemble mean DJF rainfall forecasts with PREC/L observations Source: Climate Prediction Center (http://www.cpc.ncep.noaa.gov)

22. Why South American rainfall? • Agriculture • Electricity: More than 90% produced by hydropower stations • e.g. Itaipu (Brazil/Paraguay): • • World largest hydropower plant • • Installed power: 12600 MW • • 18 generation units (700 MW each) • • ~25% electricity consumed in Brazil • • ~95% electricity consumed in Paraguay

23. Itaipu

24. Example 3: S. American rainfall anomaly composites Forecast Assimilation Obs Multi-model DEMETER: 3 coupled models (ECMWF, CNRM, UKMO) 1-month lead Start: Nov DJF ENSO composites: 1959-2001 • 16 El Nino years • 13 La Nina years ACC=1.00 ACC=0.51 ACC=0.97 ACC=Anomaly Correlation Coefficient Spatial correlation of map with obs map ACC=1.00 ACC=0.28 ACC=0.82 (mm/day)

25. DJF rainfall anomalies for 1975/76 and 1982/83 Forecast Assimilation Obs Multi-model La Nina 1975/76 ACC=0.59 ACC=-0.09 El Nino 1982/83 ACC=0.32 ACC=0.56 (mm/day)

26. DJF rainfall anomalies for 1991/92 and 1998/99 Forecast Assimilation Obs Multi-model ACC=0.32 ACC=0.04 ACC=0.08 ACC=0.38 (mm/day)

27. Brier Skill Score for S. American rainfall Forecast assimilation improves the Brier Skill Score (BSS) in the tropics

28. Reliability component of the BSS Forecast assimilation improves reliability over many regions

29. Resolution component of the BSS Forecast assimilation improves resolution in the tropics

30. Empirical model for South American rainfall Z: ASO SST Y: DJF rainfall Matrices

31. Correlation maps: DJF rainfall anomalies Empirical Multi-model Integrated • Comparable level of determinist skill • Better skill in tropical and southeastern South America

32. Mean Anomaly Correlation Coefficient Empirical Multi-model Integrated Most skill in ENSO years and forecast assimilation can improve skill

33. Brier Skill Score for S. American rainfall ENS Multi-model Integrated Empirical Forecast assimilation improved Brier Skill Score (BSS) in the tropics

34. Reliability component of the BSS Multi-model Integrated Empirical Forecast assimilation improved reliability in many regions

35. Resolution component of the BSS Integrated Multi-model Empirical Forecast assimilation improved resolution in the tropics

36. Example 4: regional rainfall downscaling Multi-model ensemble 3 DEMETER coupled models ECMWF, CNRM, UKMO 3-month lead Start: Aug NDJ Period: 1959-2001

37. South box: NDJ rainfall anomaly Multi-model - - - Observation Forecast Forecast assimilation • Forecast assimilation improves skill substantially

38. North box: NDJ rainfall anomaly Multi-model - - - Observation Forecast Forecast assimilation • Forecast assimilation improved skill marginally

39. Summary • Forecasts can be improved both by calibration and by combination • Statistical calibration and combination is analogous to data assimilation and is a fundamental and essential part of the forecasting process • (forecast assimilation) • Forecast assimilation is easy to do for normally distributed predictands such as monthly mean temperatures and seasonal rainfall: • Nino-3 probability forecasts improved – less biased and smaller spread • Equatorial SST forecasts improved in eastern and western Pacific • S. American rainfall forecasts improved in Equatorial and Southern regions • Combination can improve the resolution of the forecasts (the ability to discriminate between different observed situations) whereas calibration can improve the reliability of the forecasts • First steps towards an integrated seasonal forecasting system for South America including both empirical and coupled model predictions • EUROBRISA project will implement this system at CPTEC - Brazil

40. The EUROBRISA ProjectLead Investigator: Caio A.S. CoelhoKey Idea:To improve seasonal forecasts in S. America:a region where there is seasonal forecast skill and useful value. http://www.met.rdg.ac.uk/~swr01cac/EUROBRISA • Aims • Strengthen collaboration and promote exchange of expertise and information between European and S. American seasonal forecasters • Produce improved well-calibrated real-time probabilistic seasonal forecasts for South America • Develop real-time forecast products for non-profitable governmental use (e.g. reservoir management, hydropower production, and agriculture) EUROBRISA was approved by ECMWF council in June 2005

41. Reliability diagram (Multi-model) (oi) o (pi)

42. Direct and inverse regression Regression of obs on forecasts Regression of forecasts on obs y More natural to model uncertainty in forecasts for a given observation (ensemble spread of dots) than to model uncertainty in observations for a given ensemble forecast.  so we model the likelihood on right rather than the more common forecast calibration (MOS) approach on the left.

43. Reliability diagram (FA 58-01) (oi) o (pi)

44. Moment measure of skewness Measure of asymmetry of the distribution