James Brown, Dong-Jun Seo

1. Ensemble update conference call 02/23/09 Real Time Verification (RTV): status and plans James Brown, Dong-Jun Seo james.d.brown@noaa.gov 1

2. What do we mean by RTV? • 1. Plot historic analogs to RT forecast • Analog = similar forecast; synoptics, ….. • Show simplified EVS metrics • Get a feel for biases under similar conditions • Display bias-corrected forecast • Predict biases from past data (“ICK”) • 4. QC forecast (in EPG) before 1-3! 2

3. “Display bias-corrected forecast” 3

4. Problem definition Components of the problem: Y = {Z1,…,Zm}, real-time ensemble forecast X= observed (unknown for RT forecast) The aim is to estimate (from past data): i.e. what is unbiased (observed) prob. dist. given RT forecast? Use past relationship between forecasts and observed. 4

5. Proposed solution • Parametric post-processors not ideal for some variables (e.g. precipitation). • Can transform data (e.g. NQT), but several problems arise. • Also, correlation varies with threshold (not captured in NQT-approach). • We use non-parametric estimator that captures amount-dependence: Indicator Cokriging (ICK). Draft paper for W&F. 5

6. Pt. 1: regression model • Threshold-by-threshold correction Bias-corrected probability (at one threshold) = Climatology +Conditional adjustment (by RT forecast) 6

7. Pt. 2: parsimonious estimate “At any given threshold, we can capture 95% of variance with 9 or fewer variables (of 30).” GEFS precipitation example Total variance explained (%) 9 Total number of “ensemble members” No. of linearly independent variables 7

8. Pt. 3: fit curve to estimates c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 1.0 0.8 0.6 0.4 0.2 0.0 ICK estimate at 4th threshold, c4 ICK forecast Cumulative probability Raw forecast Smooth curve fitted through estimates (constrained quadratic spline). 0.0 100 200 300 400 500 600 Flow (CMS) 8

9. GEFS results 9

10. GEFS precipitation • Ensemble precipitation (12-hourly) from operational GEFS, 2000-2005. • Precipitation is a good test of ICK (intermittent and highly skewed). • Verified raw GEFS ensembles with ICK-corrected forecasts in Juniata, PA (MAP used as observed). • Split sample (independent) validation by rotating sample data. 10

11. Summary of results • Raw GEFS ensembles were pretty good (particularly at medium range). • But significant biases in mean, spread and higher moments. • GEFS over-forecasts small events, under-forecasts large events. • ICK forecasts were ~30% better by CRPS (key check) and more reliable. 11

12. CRPS skill of ICK (v. GEFS) “GEFS-ICK is 33% better than raw GEFS at lead hour 12.” 12

13. Conclusions and next steps • ICK shows promise • More reliable and skilful probabilities (does not currently correct “spaghetti”). • Best for cases where parametric assumptions are not good (e.g. precip.). • Well-suited to multi-model ensemble. • But: needs large sample size: look into frozen GFS hindcasts to clarify. • Need RFC to help with operations concept. 13

14. Hyperlinked slides 14

15. Example of analog display Range of observed (for analogs) Analogs (Ens. mean) RT forecast (Ens. mean) Streamflow Recent observed Past Now Future 15

16. Example of RT-EVS metric 5 4 3 2 1 0 -1 -2 -3 -4 -5 Largest member ‘Errors’ for 1 forecast RT forecast mean 90 percent. 80 percent. 50 percent. Forecast errors (forecast - observed) [inch] 20 percent. 10 percent. Smallest member Zero error line GFS-EPP precipitation (1 day total) 0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Ensemble mean forecast (increasing size) [inch] 16

17. Extra slides 17

18. 18

19. 19