NCEP GEFS 546 dm Contour

1. Similar Day Ensemble Post-Processing as Applied to Wildfire Threat and Ozone DaysMichael Erickson1, Brian A. Colle1 and Joseph J. Charney21 School of Marine and Atmospheric Sciences, Stony Brook University, Stony Brook, NY2 USDA Forest Service, East Lansing, MI

2. Ensemble Forecasting: Many Ensembles To Choose From Example from 00 UTC 10/24/2010 – 500 hPa 144 Hr Forecast NCEP GEFS 546 dm Contour CMC GGEM 546 dm Contour …And there are many other ensembles out there (i.e. ECMWF, NOGAPS, NCEP SREF, UKMET to name a few).

3. Ensemble Forecasting: A Few Ways to Look at Data Example from 00 UTC 10/24/2010 – GEFS 144 Hr Forecast Sea Level Pressure 1012 and 1028 hPa Contour NCEP SREF Mean SLP and Spread Sea Level Pressure Mean and Spread Probability of Precipitation 850 hPa Temperature Anomaly With so much data out there, can ensemble post-processing be used to benefit the operational forecaster by creating deterministic and probabilistic gridded forecasts for certain types of weather patterns? Source: http://www.emc.ncep.noaa.gov/gmb/ens/

4. SREF+SBU 2-m Temperature Bias > 298 K Between 1200-0000 UTC Diurnal Mean Error – SREF+SBU Bias by Member > 298 K • Using ensemble output directly may be misleading, since ensembles are frequently underdispersed with large surface model biases. • Although ensemble post-processing methods are growing in sophistication, the sensitivity of biases to the synoptic flow pattern is not well known. Caveats

5. Do model biases vary with the ambient surface weather conditions (i.e. on days with high fire threat and high ozone)? • Does applying a similar day approach to ensemble post-processing improve deterministic and probabilistic forecasts? • Are there any dominant atmospheric flow patterns during these anomalous events that are related to model biases? • Can similar day ensemble post-processing be used to create simple, helpful and skillful forecasts in operations? Questions to Be Addressed

6. Analyzed the Stony Brook University (SBU) and NCEP Short Range Ensemble Forecast (SREF) system for 2-m temperature and 10-m wind speed. • The Automated Surface Observing System (ASOS) are used as verifying observations from 2007-2009 over a subset of the Northeast. Region of Study Methods and Data Region of Study 00 UTC SBU 13 Member Ensemble • Consists of 7 MM5 and 6 WRF members run at 12 km grid spacing within a larger 36 km nest. • Variety of ICs (GFS, NAM, NOGAPS, CMC), microphysical, convective and PBL schemes. Verification Domain 21 UTC NCEP SREF 21 Member Ensemble • 10 ETA, 5 RSM, 3 WRF-NMM, and 3 WRF-ARW. • IC’s perturbed using a breeding technique.

7. Bias Correction Methods • Running Mean Bias Correction: Determine bias over training period and subtract it from forecast (Wilson et al. 2007): • CDF Bias Correction: Adjust model CDF to the observed CDF for all forecast values (Hamill and Whitaker 2005), then elevation and landuse. • Wind speed was bias corrected with the CDF method, temperature used the running mean method. CDF Bias Correction Example CDF For Model and Observation

8. Exploring Model Bias on Hazardous Weather Days High Fire Threat Classification • Used the Fire Potential Index (FPI) from the Woodland Fire Assessment System (WFAS) between 2007-2009. • A fire threat day must have 10% or greater of the domain reach 50 FPI while the remainder of the domain exceeds 25 FPI. High Ozone Threat Classification • A high ozone day must have 10% of AIRNow stations with an Air Quality Index (AQI) > 60 ppb, while the remainder of the domain > 30 ppb. Additional Details • Explored the impact of training period on post-processing for high fire threat and ozone days: • Sequential Training – Used the most recent 14 consecutive days. • Conditional Training – Used the most recent 14 similar days. • Analyzed daytime model output (1200-0000 UTC) for ensembles initialized the day of and the day before the hazardous weather event (i.e. SBU model hours 12-24 and 36-48).

9. Bias Correction Methods for Fire Threat Days - Temperature ME per Model For Temperature MAE per Model For Temperature • Sequential bias correction with temperature still has an average ensemble mean bias of -0.8 K, which is removed when using conditional bias correction. • MAE is also improved for almost every ensemble member and the ensemble mean.

10. Bias Correction Methods for Fire Threat Days – Temperature > 298 K Raw Warm Season Raw Warm Season Raw Fire Threat Days Seq. Bias Cor. Fire Threat Days Cond. Bias Cor. Fire Threat Days • Spatially, the negative temperature bias on high fire threat days is found in every station. • Conditional bias correction removes the negative temperature bias and reduces the spread of the biases.

11. Bias Correction Methods for Fire Threat Days – Wind Speed ME per Model For Wind Speed MAE per Model For Wind Speed • High fire threat days have a smaller positive wind speed bias than the warm season average. • As with temperature, conditional bias correction removes the bias and improves MAE.

12. Bias Correction Methods for High Ozone Days – Wind Speed > 2.6 m/s Raw Warm Season Raw High Ozone Days Seq. Bias Cor. High Ozone Days Cond. Bias Cor. High Ozone Days • Wind speed model biases on high ozone days are also less positively biased than the warm season average. • Spatially, the CDF bias correction removes most of the bias, although not as effectively as the additive bias correction with temperature.

13. Reliability Plots for Conditional Bias Correction High Ozone Days – Wind Speed • Even after conditional bias correction, reliability plots still reveal a lack of probabilistic skill, indicative of a strong amount of underdispersion. • Therefore, additional post-processing is necessary.

14. Bayesian Model Averaging (BMA) • Bayesian Model Averaging (BMA, Raftery et al. 2005) calibrates ensemble forecasts by estimating: • Weights for each ensemble member. • The uncertainty associated with each forecast. • 10 members were selected from the SBU/SREF system with a training period of 28 days. • 5 best SBU members (in terms of MAE) for each PBL scheme. • 5 control SREF members. • The same model hours and training method used in bias correction is also used for BMA. • Parameters estimated using a MCMC method developed by Vrugt et al. (2008). Sample BMA PDF For Wind Speed The BMA derived distribution Members Have Varying Weights

15. Reliability Plots for Conditional Bias Correction and BMA High Ozone Days – Wind Speed • BMA improves probabilistic results (i.e. reliability corresponds closer to the 1:1 line) for high ozone wind speed results. • However, it is important to evaluate ensemble dispersion on the average.

16. Rank Histograms of Temperature for Hazardous Weather Days Sequential Training – Fire Threat Days Conditional Training – Fire Threat Days Conditional Training – High Ozone Days Sequential Training – High Ozone Days • BMA greatly improves ensemble underdipsersion, but can not correct any lingering bias as a result of using sequential training.

17. Brier Skill Scores - Conditional and Sequential BMA Referenced Against Sequential Bias Correction – Temperature Fire Threat Days - Temperature High Ozone Days - Temperature • Brier Skill Scores (BSS) indicate probabilistic benefit with values greater than zero. • Since BSS is usually > 0, BMA improves the ensemble regardless of training. • In many cases, conditional BMA performs better than sequential BMA, with statistically significant improvement on high fire threat days.

18. Brier Skill Scores - Conditional and Sequential BMA Referenced Against Sequential Bias Correction – Wind Speed Fire Threat Days – Wind Speed High Ozone Days – Wind Speed • Unlike temperature, the difference between conditional and sequential BMA is not statistically significant.

19. BMA in Operational Forecasting – High Fire Threat Example Ensemble Mean Forecast on April 25th 2008 Conditional BMA Mean Forecast Sequential Bias Corrected Mean Forecast Conditional Bias Corrected Mean Forecast Raw Mean Ensemble Forecast Observation • BMA can be used to generate a deterministic forecast spatially over the entire region.

20. BMA in Operational Forecasting – High Fire Threat Example Probability > 287 K on April 25th 2008 Conditional BMA Forecast Conditional Bias Corrected Forecast Sequential Bias Corrected Forecast Observation Raw Ensemble Forecast • BMA can be used to generate probabilistic forecasts for critical thresholds that are typically more accurate than bias correction or the raw ensemble.

21. Sensitivity of Ensemble Performance to Member Selection Warm Season – Conditional Bias Correction; No BMA • A key assumption when running BMA was that the 5 SBU and 5 SREF members selected was a good choice. • This assumption is tested by comparing the 10 SBU/SREF ensemble used previously (B10) to a randomly selected 10 member ensemble (R10) 1000 times. Hazardous Weather Days – Conditional Bias Correction and BMA • The sensitivity of BMA performance to member selection is tested by rerunning BMA with 2 ensembles consisting of 10 members each: • The same 5 SBU and 5 SREF member ensemble used earlier (B10-BMA). • 5 randomly selected SBU and 5 randomly selected SREF members (R10-BMA). • The benefits of combining the SBU + SREF is tested by creating 3 ensembles from B10. • 5 best SBU members (B5-SBU-BMA). • 5 best SREF members (B5-SREF-BMA). • 2.5 randomly selected best SBU and 2.5 randomly selected best SREF (B5-ALL-BMA).

22. Sensitivity of Ensemble Performance to Member Selection – Warm Season – Bias Correction (No BMA) Spread: B10 Minus R10 Ensemble Spread: B10 Minus R10 Ensemble BSS: B10 Referenced Against R10 BSS: B10 Referenced Against R10 MAE: B10 Minus R10 Ensemble MAE: B10 Minus R10 Ensemble Temperature Wind Speed • The B10 ensemble has lower MAE (i.e. more skill) than R10, but it also has less ensemble spread (i.e. is more underdispersed) for temperature and wind speed. • This underdispersion results in B10 having less probabilistic skill than R10 with wind speed

23. Brier Skill Scores – Comparison Between Best Ensemble and Randomly Generated Ensembles- Temperature Fire Threat Days High Ozone Days • B10-BMA performs better probabilistically than R10-BMA. Since this was not the case with B10 and R10, BMA can correct for underdispersion if the ensemble has deterministic skill.

24. Brier Skill Scores – Benefits of Adding the SBU and SREF Ensembles - Temperature Fire Threat Days High Ozone Days • The combined SBU and SREF ensemble (5B-ALL-BMA) performs better than the SBU (5B-SBU-BMA) and SREF (5B-SREF-BMA) separately on high fire threat days. Results with high ozone days are mixed, with no clear benefit or loss with using 5B-ALL-BMA.

25. Environmental Modes on High Fire Days – 500 hPa Height Anomaly • High fire threat days may be associated with a few consistent large-scale atmospheric flow regimes. • In order to examine this, the North American Regional Reanalysis (NARR, 32 km grid spacing) 3-hour composites were gathered on high fire threat days. • Atmospheric modes on high fire threat days were captured using Empirical Orthogonal Functions (EOF) analysis on 500 hPaheight anomalies, and the dominant Principal Components (PCs) were correlated to the 2-m temperature model bias.

26. Environmental Modes on High Fire Days – 500 hPa Height Anomaly 500 hPa Anomaly EOF 1 - 4

27. Conclusions • High fire threat and ozone days have cooler 2-m temperature and less windy 10-m wind speed model biases compared to the warm season average. • Conditional post-processing is better than sequential training at removing biases and calibrating the ensemble for high fire threat and ozone days. • Results with a similar day approach suggests that analog post-processing could be used to create unbiased and skillful gridded operational forecasts. • Similar day post-processing could be extended to benefit forecasts of high impact events (Nor‘easters, cold snaps, heat waves, wind energy, etc). • However, it is not immediately obvious how to implement a large ensemble of unique members with BMA. Combining the SBU and SREF ensembles and picking skillful members (in terms of MAE) is frequently better than randomly selecting members. • Preliminary EOF results suggest the presence of 500 hPa flow regimes on high fire threat days that are correlated to the 2-m temperature model biases. Exploring the cause for these biases could lead to an improvement in the model parameterized physics.