Outline

1. Preliminary Skill Comparisons AmongIRI-Multi-Model Ensemble-,CDC-CCA- Hindcasts andCPC-Official Seasonal Temperature ForecastsEd O’Lenic (CPC), Wesley Ebisuzaki (CPC), Marty Hoerling (CDC), Lisa Goddard (IRI)28th Climate Diagnostics and Prediction WorkshopReno, NevadaOctober 20, 2003

2. Outline • US-average skill of categorical and probabilistic seasonal forecasts of US surface temperature is examined. • Hindcasts of IRI multi-model ensemble (MM) from 1998-2003 and CDC-CCA (CCA) from 1995 to 2003 are compared with CPC-Official forecasts (OFF) from 1995 to 2003. • This comparison serves as a sanity check on the verification system and points out advantages and disadvantages of the several techniques. • Among the recommendations are: *CPC forecasts may be too conservative (encouraged by the skill metric we use) in areal coverage and in probabilities, *MM and CCA probabilities appear too large, *MM and CCA categorical forecasts may be better than 1/3 in cl regions, *MM and CCA are sufficiently skillful to serve as a first-guess forecast for CPC’s seasonal forecast system.

3. Cautionary Notes • CPC (OFF) forecasts are the original operationally- issued forecasts, while those from IRI (MM) and CDC (CCA) are hindcasts. • We do not know for certain whether these hindcasts would be as good as they are if they were issued operationally. • MM occasionally predicts CL (EC), CCA does not, OFF uses CL alot. • MM and CCA predict the near normal category, OFF almost never does. • The sample size of these datasets is small and results are likely not statistically significant. • CL and EC are used interchangeably.

4. Model Descriptions • MM – IRI’s fully-automated multi-model ensemble of GCM forecasts. Data begins in February, 1998. Hindcasts. • CCA – CDC’s fully-automated technique which uses CCA to predict what a combination of several models would predict as a function of sst. Forced by presisted ssts. Data begins January, 1995. Hindcasts. • OFF – CPC’s subjective official forecasts. Based on statistical and dynamical forecast tools, including MM and CCA, along with Constructed Analogs, individual model forecasts, Trend, Soil Moisture and ENSO and ENSO-neutral composites. Data from Jan 1995. Operational Forecasts. • These are available at: http://www.cpc.ncep.noaa.gov/products/predictions/90day/tools/briefing/index.pri.html

5. Skill Scores • Non-EC stations: s=((c-e)/(t-e))*100 • All Stations : s=((c+(1/3)*cl-e)/(t-e))*100 c = # stations correct e = # stations correct by chance t = # stations in total cl= # stations predicted equal chances • rpss = 1- rps/rpsclimate • rps = (1/(M-1)) 3[(3pu) – (3ou)] M m m 2 M=1 u=1 u=1

6. TEMP PREC A sample forecast: JFM 1999 CCA MM OFF OBS

7. MM cl CCAcl 95 96 97 98 99 00 01 02 03 95 96 97 98 99 00 01 02 03 MM No-cl CCA No-cl 95 96 97 98 99 00 01 02 03 95 96 97 98 99 00 01 02 03

8. 0.06 vs 0.13 OFF (+), better 13 MM (+), better 17 OFF (+) MM (-) 6 6 MM (+) OFF (-) 16 6

9. 0.12 vs 0.12 OFF (+), better 22 MM (+), better 8 OFF (+) MM(-) 7 3 MM (+) OFF(-) 15 10

10. 0.08 vs 0.12 OFF (+), better 18 OFF (+) CCA (-) 16 CCA (+), better 28 15 CCA (+) OFF (-) 20 3

11. 0.17 vs 0.12 OFF (+), better 33 CCA (+), better 14 OFF (+) CCA(-) 16 7 CCA (+) OFF(-) 20 10

12. RPSS=0.02 RPSS=0.00 CCA OFF CCA and MM probabilistic forecasts appear too liberal, OFF too conservative. RPSS of all have declined over the last 2 years RPSS=0.00 MM

13. Conclusions, Recommendations • CPC uses non-Cl T skill as its metric. This encourages forecasting small areas. This is bad only if generally increased area is justified. We did better before 1997. Is a better set of skill masks needed? • Purely numerical MM does as well in non-Cl T forecasts as combined statistical/numerical CPC. • CPC does better than CCA in non-Cl T forecasts. • Both CCA and MM do better than CPC in Cl T forecasts. This appears to indicate CCA and MM score better than 1/3 in those areas. • CCA Cl T forecasts have high skill when CPC skill is low. • Should areas of EC should be reduced in CPC forecasts (CPC too conservative)? • The decline of skill in all the tools over the last 2 years indicates reduced predictability. • All three probabilistic forecast systems can be improved by calibration. • Differences among forecast skills likely not statistically significant. • MM and CCA should be combined and used by CPC as a first guess.