1 / 34

Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I

Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I. LAST YEAR / THIS YEAR:. Last Year: Ridge Regression as a Consolidation Method, to yield, potentially, non-equal weights. This Year: see new Posters on Consolidation by

gradyh
Download Presentation

Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I

  2. LAST YEAR / THIS YEAR: • Last Year: Ridge Regression as a Consolidation Method, to yield, potentially, non-equal weights. • This Year: see new Posters on Consolidation by -) Malaquias Pena (methodological twists and SST application) and -) Peitao Peng (application to US T&P) • Last Year: Conversion to pdf as per “Kernel method” (Dave Unger). This Year: Time series approach is next.

  3. Does the NCEP CFS add to the skill of the European DEMETER-3 to produce a viable International Multi Model Ensemble (IMME) ?“Much depends on which question we ask” Input by Suranjana Saha and Ake Johansson is acknowledged.

  4. DATA and DEFINITIONS USED • DEMETER-3 (DEM3) = ECMWF + METFR + UKMO • CFS • IMME = DEM3 + CFS • 1981 – 2001 • 4 Initial condition months : Feb, May, Aug and Nov • Leads 1-5 • Monthly means

  5. DATA/Definitions USED (cont) • Deterministic : Anomaly Correlation • Probabilistic : Brier Score (BS) and Rank Probability Score (RPS) • Ensemble Mean and PDF • T2m and Prate • Europe and United States Verification Data : • T2m : Fan and Van den Dool • Prate : CMAP

  6. BRIER SCORE FOR 3-CLASS SYSTEM 1. Calculate tercile boundaries from observations 1981-2001 (1982-2002 for longer leads) at each gridpoint. 2. Assign departures from model’s own climatology (based on 21 years, all members) to one of the three classes: Below (B), Normal (N) and Above (A), and find the fraction of forecasts (F) among all participating ensemble members for these classes denoted by FB, FN and FA respectively, such that FB+ FN+FA=1 . 3. Denoting Observations as O, we calculate a Brier Score (BS) as : BS={(FB-OB)**2 +(FN-ON)**2 + (FA-OA)**2}/3, aggregated over all years and all grid points. {{For example, when the observation is in the B class, we have (1,0,0) for (OB, ON, OA) etc.}} 4. BS for random deterministic prediction: 0.444 BS for ‘always climatology’ (1/3rd,1/3rd,1/3rd) : 0.222 5. RPS: The same as Brier Score, but for cumulative distribution (no-skill=0.148)

  7. Number of times IMME improves upon DEM-3 :out of 20 cases (4 IC’s x 5 leads): “The bottom line” “ NO consolidation, equal weights, NO Cross-validation”

  8. Anom.Corr US T2m February 1982-2002 lead 3 November start

  9. Anom.Corr US T2m February 1982-2002 lead 3 November start

  10. Cross Validation (CV) • Why do we need CV? • Which aspects are CV- ed: a) systematic error correction ( i) the mean and ii) the stand.dev) and b) weights generated by Consolidation • How?? CV-1, CV-3, CV-3R • Don’t use CV-1!. CV-1 malfunctions for systematic error correction in combination with internal* climatology and suffers from degeneracy when weights generated by Consolidation are to be CV-ed. • *Define internal and external climatology

  11. Last year wrt OIv2 1971-2000 climatology

  12. Last year wrt OIv2 1971-2000 climatology

  13. w1 w2 w3 w4 nov lead 3 check .35 .04 .27 .35 11 3 1981 left out (and 2 others) nov lead 3 check .36 .00 .29 .35 11 3 1982 left out (and 2 others) nov lead 3 check .29 .02 .33 .36 11 3 nov lead 3 check .40 .04 .31 .36 11 3 nov lead 3 check .38 -.01 .28 .34 11 3 nov lead 3 check .37 .00 .27 .33 11 3 nov lead 3 check .35 -.01 .25 .40 11 3 nov lead 3 check .31 .01 .29 .43 11 3 nov lead 3 check .28 .01 .31 .39 11 3 nov lead 3 check .38 .02 .30 .32 11 3 nov lead 3 check .36 .00 .39 .32 11 3 nov lead 3 check .31 .03 .29 .39 11 3 nov lead 3 check .45 -.01 .23 .35 11 3 nov lead 3 check .37 .00 .31 .41 11 3 nov lead 3 check .35 .03 .28 .40 11 3 nov lead 3 check .33 .02 .36 .35 11 3 nov lead 3 check .42 .01 .33 .31 11 3 nov lead 3 check .33 .04 .31 .42 11 3 nov lead 3 check .33 .00 .29 .35 11 3 nov lead 3 check .40 .02 .31 .35 11 3 nov lead 3 check .33 .00 .24 .38 11 3 2001 left out (and 2 others) Feb forecast has high co-linearity. Model 2 has high –ve weights for unconstrained regression

  14. Overriding conclusion  With only 20+ years of hindcasts it is hard for any consolidation to be much better than equal weight MME. (Give us 5000 years.) ‘Pooling’ data helps stabilize weights and it increases skill, but is it enough?  20+ years is a problem even for CV-ed systematic error correction.

  15. Further points of study • The nature of climatology (= control in verification), external, internal, fixed • Cross Validation method not settled • The Many Details of Consolidation as per Ridge Regression • Conversion to pdf can be done in very many different ways (including 3-class BS minimization, logistic regression, ‘count method’, Kernels)

  16. Forecast Consolidation at CPC – Part 2Ensembles to Probabilities David Unger / Huug van den Dool Acknowledgements: Dan Collins, Malaquias Pena, Peitao Peng

  17. Objectives • Produce a single probabilistic forecast from many tools - Single value estimates - Ensemble sets • Utilize Individual ensemble members - Assume individual forecasts represent possible realizations - We want more than just the ensemble mean • Provide Standardized Probabilistic output - More than just a 3-class forecast

  18. Kernel Characteristics

  19. Kernel Characteristics Unequal Weighting

  20. Ensemble Regression • A regression model designed for the kernel smoothing methodology - Each member is equally likely to occur - Each has the same conditional error distribution in the event it is closest to the truth. F= Forecast, σF = Forecast Standard Deviation Obs=Observations, σObs= Standard Deviation of observations R=Correlation between individual ensemble members and the observations Rm = Correlation between ensemble mean and observations a1 , a0 = Regression Coefficients, F = a0 + a1 F

  21. Time series estimation • Moving Average, Let X11 Be the 10-year running mean known on year 11. N=10 X11 = 1/N(x1+x2+x3+x4+x5+x6+x7+x9+x9+x10) X12 = X11 + 1/N(x11-x1) XY+1 = XY + 1/N(xY+1-xY-10) • Exponential Moving Average (EMA), α = 1/N X12 = X11 + α(x11- X11) XY+1 = (1- α)XY + αxY+1

  22. Adaptive Ensemble Regression EMA estimates • F • F2 • (Obs) • (Obs)2 • F (Obs) • Fm2 • (F-Fm)2

  23. Trends • Adaptive Regression “learns” recent bias, and is very good in compensating. • Most statistical tools also “learn” bias, and adapt to compensate. Steps need to be taken to prevent doubling bias corrections.

  24. Trends (Continued) • Step 1. Detrend all models and Obs. F = F – F10Obs = Obs – Obs10 F10 , Obs10 = The EMA approximating a 10-year mean • Step 2. Ensemble Regression Final forecast set, F are anomalies. • Step 3. Restore the forecast. A) F = F + F10 We believe the OCN trend estimate B) F = F + C30: C30 = 30-year (1971-2000) Climatology We have no trust in OCN. C) F = F +C30+ROCN (F10 –C30) : ROCN= Correlation ( F10,Obs) Trust but verify.

  25. Weighting • The chances of an individual ensemble member being “best” increases with the skill of the model. • The kernel distribution represents the expected error distribution of a correct forecast.

  26. Final Forecast • Consolidated Probabilities are the area under the PDF within each of three (Below, Near, Above median ) categories. Call for ABOVE when the P(above)>36% and P(Below) <33.3% Call for BELOW when P(Below > 36%) and P(Above) < 33.3% White area = Equal Chances (We don’t as yet trust our Near Normal percentages)

  27. Performance Tools 1995 – 2005. • CFS – 15 members hindcast All Members weighted equally with combined area equal to the tool weighting • CCA - Single Valued Forecast Hindcasts from Cross Validation • SMLR - Single valued forecast – Hindcasts from Retroactive Real-time Validation • OCN incorporated with EMA rather than 10-year box car average.

  28. Performance (Continued) • First Guess EMA parameters provided by CCA, SMLR Statistics 1956-1980. • CFS spinup 1981-1994 Validation Period All Initial times, 1-month lead, Jan 1995-Dec 2005 (11 Years)

  29. Performance (Continued) • Official Forecast: Hand-drawn, probabilities in 3 classes. PoE obtained from Normal distribution, Standard deviation based on tool skills) • CCA+SMLR: A Consolidation of the two Statistical Forecasts, equally Weighted. • CFS: A Consolidation of 15 CFS ensembles, equally weighted. • CFS+CCA+SMLR Wts: A Consolidation of CCA, CFS, and SMLR, weighted by R/(1-R) for each of the three tools. 15 CFS members each given 1/15th of the CFS weight. Also known as “All” • All – Equal Wts: CCA, SMLR and the 15 CFS members combined are given equal Weights. • All – No Trend. Anomalies applied to 30-year mean.

  30. Performance CRPSS RPSS - 3 HSS Bias (C) % Cover CCA+SMLR CFS CFS+CCA+SMLR, Wts. All – No Trend All – Equal Wts. Official

  31. CRPS Skill Scores: Temperature 1995 – 2005 Skill .10 .05 .01 All CCA+SMLR 1-Month Lead, All initial times CFS Official

  32. CRPS Skill Scores / % cover: Temperature 1995 – 2005 Skill .10 .05 .01 Trends No Trends CRPSS 1-Month Lead, All initial times %cover

  33. The Winter Forecast Skill Weighting Equal Weighting

  34. Conclusions • Weighting is not critical – within reason. • Consolidation outperforms component models . • Getting the trends right is essential. • CFS + Trend consolidation provides an accurate continuous distribution.

More Related