Using reforecasts for probabilistic forecast calibration

1. NOAA Earth System Research Laboratory Using reforecasts for probabilistic forecast calibration Tom Hamill & Jeff Whitaker NOAA Earth System Research Lab, Boulder, CO tom.hamill@noaa.gov

2. Reforecasts? • A hindcast, a numerical prediction for a date in the past using the model and data assimilation system that is currently operational. • Uses: • (1) Post-processing of ensemble weather and climate forecasts; correcting for systematic bias, spread deficiencies, downscaling. Crucial for implementing reliable uncertainty forecasts in NWS. • (2) Data assimilation: first-guess forecasts corrected by observations; forecasts assumed unbiased; reforecasts can help adjust to make sure they are. • (3) Diagnosing model errors: sometimes model deficiencies aren’t obvious without large sample.

3. NOAA’s 1st-generation reforecast data set • Model: T62L28 NCEP GFS, circa 1998 • Initial States: NCEP-NCAR Reanalysis II plus 7 +/- bred modes. • Duration: 15 days runs every day at 00Z from 19781101 to now. (http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2). • Data: Selected fields (winds, hgt, temp on 5 press levels, precip, t2m, u10m, v10m, pwat, prmsl, rh700, heating). NCEP/NCAR reanalysis verifying fields included (Web form to download at http://www.cdc.noaa.gov/reforecast). Data saved on 2.5-degree grid. • Experimental precipitation forecast products: http://www.cdc.noaa.gov/reforecast/narr .

4. Reforecastingadvantages & disadvantages • Advantages: • Very large gains in skill and reliability from removing systematic errors (demonstrated later). • A real pathway to the NWS providing objective, reliable, skillful probabilistic forecasts without a lot of human intervention (no fleet of forecasters working on probabilistic IFPS). • Disadvantages • Computationally expensive, especially if model is changing frequently. • Not getting forecast improvement directly through improving the model.

5. Application: NCEP/CPC’s 6-10 day outlook Map of probabilities of above / below / near normal. 33 percent probability assumed in near normal unless above or below > 67 percent.

6. Comparison against NCEP / CPC forecasts at 155 stations, 100 days in winter 2001-2002 temperature forecasts Reforecast calibrated Week-2 forecasts more skillful than operational NCEP/CPC 6-10 day, which was based on human blending of NCEP, ECMWF, other tools. precipitation forecasts

7. Reforecast-based example: floods causing La Conchita, California landslide, 12 Jan 2005 week-2 from reforecast 6-10 day from reforecast

8. Reforecast model brought back into production and used operationally at NCEP/CPC because of the usefulness of reforecast products. Also working on transition of products to NCEP/HPC

9. Application: downscaled precipitation forecasts using analog technique On the left are old forecasts similar to today’s ensemble- mean forecast. The data on the right, the analyzed precipitation conditional upon the forecast, can be used to statistically adjust and downscale the forecast. Analog approaches like this may be particularly useful for hydrologic ensemble applications, where an ensemble of realizations is needed.

10. Downscaled analog probability forecasts

11. Verified over 25 years of forecasts; skill scores use conventional method of calculation which may overestimate skill (Hamill and Juras 2006).

12. Tornado probability forecasting forecast wind shear and instability were used as predictors in an analog approach.

13. ECMWF’s reforecast experiments ECMWF got excited by our results and produced a test reforecast data set to see if they would get a big forecast improvement even with their much-improved ensemble forecast system. • Model: 2005 version of ECMWF model; T255 resolution. • Initial Conditions: 15 members, ERA-40 analysis + singular vectors • Dates of reforecasts: 1982-2001, Once-weekly reforecasts from 01 Sep - 01 Dec, 14 weeks total. So, 20y  14w ensemble reforecasts = 280 samples. • Data obtained by NOAA / ESRL : T2M and precipitation ensemble over most of North America, excluding Alaska. Saved on 1-degree lat / lon grid. Forecasts to 10 days lead.

14. ECMWF, raw and post-processed Note: 5th and 95th %ile confidence intervals very small, 0.02 or less

15. ECMWF, raw and post-processed In this metric, calibrated 4-5 day forecasts now as skillful as uncalibrated 1-day forecast. Note: 5th and 95th %ile confidence intervals very small, 0.02 or less

16. Precipitation: 5-mm reliability diagrams(~90-km forecasts verified against 32-km North American regional reanalysis) horizontal lines indicate distribution of climatology error bars from block bootstrap Raw forecasts have poor skill in this strict BSS; much improved with calibration

17. Precipitation skill with weekly and 30-day training data sets Compared use of the once-weekly  20-year reforecast data set to calibration using only the past 30 days of forecasts. Substantial benefit of weekly reforecasts relative to 30-day training data sets, especially at high thresholds; for the more rare heavy precipitation events, a longer training data set is needed.

18. ECMWF newsletter, Autumn 2008: Reforecasts operational at ECMWF

19. NOAA and new reforecasts? • Climate Forecast System Reanalysis and Reforecast (operational 2010-2011) • 1 reforecast member per day; T126 model used for climate forecasts. • GEFS: Global ensemble forecast system; plan to do 1 reforecast member per day in real time, e.g., on 1 Dec 2009, do 1 Dec 2008, 2007, etc.

20. Reforecast issues How many members? Results with ECMWF data set suggest additional benefit from more members.

21. Reforecast issues.Which method of calibration? NCEP was hopeful that a new calibration method, “Bayesian Processor of Forecasts” would allow them to do improved calibrations with small training data sets, lessening the need for reforecasts. Recent ESRL/PSD research has demonstrated serious problems with the proposed algorithm relative to ones that have been previously demonstrated to be effective.

22. Other reforecast issues • Which calibration method is best for other important parameters? (clouds, precipitation amount/type, winds, severe weather, etc.) • What is NOAA’s long-term strategy for production of probabilistic forecasts? How does reforecasting fit with the overall strategy for probabilistic forecasting? • Who does what? ESRL or NCEP to compute reforecasts? NCEP or MDL to post-process? Technique development at ESRL, NCEP, and/or MDL? • Best configuration of reforecast data set? • Reforecasts computed in real-time, with evolving model, few members (NCEP’s preferred approach for GEFS), or • Fixed model, large reforecast data set, computed all at once, and used for many years? • Does the configuration of the best reforecast data set vary with weather parameter? For example, do hydrologists need more data than is needed for temperature calibration?

23. FY11 Alternative (Climate goal) • Develop next-generation reforecast with modern NCEP model, based on CFSRR reanalyses. • Examine reforecast ensemble size issues - how much of the benefit is possible with reduced size data set (fewer members, reforecasts every x days). • Compare against evolving NCEP ensemble system with bias corrections from 1-member reforecast. • Develop suite of new experimental products. • Etc. • NWS did not support making this climate activity part of FY11 core; it’s “above core” currently.

24. Conclusions • Reforecasts shown to aid in calibration of forecasts for a wide variety of applications. • Still a large benefit from forecast calibration, even with state-of-the-art ECMWF forecast model. • Many remaining issues to be explored, and renewed importance given recent emphasis on uncertainty forecasting in NWS. • Still working on securing stable funding & CPU time for next-generation reforecasts.