Comparing Postprocessing Approaches to Calibrating Operational River Discharge Forecasts

1. Comparing Postprocessing Approaches to Calibrating Operational River Discharge Forecasts Tom Hopson1 Peter Webster, EAS-Georgia Tech Andy Wood, CBRFC-NOAA 1

2. Questions • How are forecasting errors partitioned (i.e. “lowest hanging fruit”)? • Capturing (reducing?) hydrologic model errors • Best algorithms (KNN, QR)? • Impact of # of hindcasts? • Optimal point to apply post-processing algorithms?

3. Context: CFAB Project PI Peter Webster, Georgia Tech (pjw@eas.gatech.edu) Purpose: provide flood forecasts of the Ganges and Brahmaputra rivers for Bangladesh, operational 2003-ongoing Partners: USAID, CARE, ECMWF, Bangladesh’s Meteorology Dept and Flood Forecasting Warning Centre (FFWC), NASA-TRMM, NOAA-CMORPH

4. Daily Operational Flood Forecasting Sequence Reference: Hopson, T. M., and P. J. Webster, 2010: A 1–10-Day Ensemble Forecasting Scheme for the Major River Basins of  Bangladesh: Forecasting Severe Floods of 2003–07. J. Hydrometeor., 11.

5. Precipitation Forecast Bias Adjustment • done independently for each forecast grid • (bias-correct the whole PDF, not just the median) Model Climatology CDF “Observed” Climatology CDF Pmax Pmax Precipitation Pfcst Padj 25th 50th 75th 100th 25th 50th 75th 100th Quantile Quantile In practical terms … ranked forecasts ranked observations 0 1m 0 1m Precipitation Precipitation

6. Brahmaputra Discharge Forecasts for 2007 Model driven with ECMWF 51-member ensemble => Need to account for hydrologic model (and observed precip) errors!

7. Skill Scores • Single value to summarize performance. • Reference forecast - best naive guess; persistence, climatology • A perfect forecast implies that the object can be perfectly observed • Positively oriented – Positive is good

8. Skill Score Verification RMSE Score CRPS Score RMSE Skill Score CRPS Skill Score Reference Forecasts: green – climatology red -- persistence Reference Forecasts: green – climatology red -- persistence

9. Assessment of Hydrologic Forecast Error Sources Total stddev Hydro model/obs precip stddev Weather variable stddev Recall for errors of 2 variables v1 and v2added in quadrature:

10. Final flood forecast “calibration” or “post-processing” “bias” obs Forecast PDF Probability Probability Forecast PDF obs “spread” or “dispersion” calibration Flow rate [m3/s] Flow rate [m3/s] • Post-processing has corrected: • the “on average” bias • as well as under-representation of the 2nd moment of the empirical forecast PDF (i.e. corrected its “dispersion” or “spread”) • Additional Goals: • probability distribution function “means what it says” (flat rank histogram) • daily variation in the dispersion directly relate to changes in forecast skill • produce PDF that has as much information content as possible (i.e. “narrow”)

11. 1 PDF Step 1: generate discharge ensembles from precipitation forecast ensembles (Qp): Probability 1/51 Qp [m3/s] Step 2: a) generate multi-model hindcast error time-series using precip estimates; b) conditionally sample and weight to produce empirical forecasted error PDF: a) 1000 forecast horizon b) 1 Residuals PDF [m3/s] time => Residual [m3/s] -1000 1000 -1000 1 Step 3: combine both uncertainty PDF’s to generate a “new-and-improved” more complete PDF for forecasting (Qf): Probability Qf [m3/s] Producing a Reliable Probabilistic Discharge Forecast

12. Our approach: Quantile Regression (QR) Benefits Less sensitivity to outliers Works with heteroscedastic errors Optimally fit for each part of the PDF 4) “flat” rank histograms

13. Results: time series Uncorrected KNN QR 5-day lead-time 10-day lead-time => QR appears to be more stable than KNN

14. Rank Histograms Uncorrected KNN QR => Increased stability of QR reflected in slightly more consistent rank histograms

15. CRPS Skill Score Comparisons of: 1) Post-processing; 2) Data size Referenced to Uncorrected Forecasts KNN QR 125pts 1000pts

16. RMSE Skill Score Comparisons of: 1) Post-processing; 2) Data size Referenced to Uncorrected Forecasts KNN QR 125pts 1000pts Points: 1) degradation of mean for small error corrections (short lead-times), especially for QR; 2) KNN appears to provide an actual forecast correction for long lead-times; 3) But KNN less stable for small data sets

17. Summary • Investigated different algorithms (KNN and QR) and training data size 125-1000pts) to post-processing ensemble discharge forecasts in the operational setting of the Brahmaputra river entering into Bangladesh • Forecasting system post-processes separately precipitation forecasting error from hydrologic model (and observed precipitation) error • For the Brahmaputra catchment (with a time of concentration roughly 7 days), impact of precipitation forecasting error inconsequential out to 4 days • Under this particular forecasting approach, both KNN and QR provide reliable (flat rank histograms) and sharp (roughly 70-80% improvements over persistence) final PDFs • Post-processing inflates the dispersion (2nd moment) of the PDFs, which increases the sensitivity of the PDFs to degradation of their 1st moment skill • Comparisons of KNN and QR show greater stability in the QR approach, but possible greater sensitivity of KNN to forecasting hydrologic model errors