Israel Jirak, Steve Weiss, and Chris Melick Storm Prediction Center

1. Evaluation and Comparison of Multiple Convection-Allowing Ensembles Examined in Recent HWT Spring Forecasting Experiments Israel Jirak, Steve Weiss, and Chris Melick Storm Prediction Center WoF Workshop, April 3, 2014

2. Convection-Allowing EnsemblesOverview • Convection-allowing ensembles (~4-km grid spacing) can provide important information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. on the outlook to watch scale • These ensembles will play an important role in the ability of SPC to provide a more continuous flow of probabilistic hazard information in support of WoF 2 March 2012 29 June 2012 24-h neighborhood prob UH ≥25 m2/s2 24-h ensemble max 10-m Wind Speed

3. Convection-Allowing EnsemblesHistory • An experimental real-time Storm-Scale Ensemble Forecast (SSEF) system has been produced for the NOAA Hazardous Weather Testbed (HWT) Spring Forecasting Experiment (SFE) by OU CAPS since 2007 through CSTAR funding • Comprised of 4-km convection-allowing WRF and ARPS members: • 2007: 10 members; 2008: 10 members; 2009: 20 members; 2010: 26 members (full CONUS to 30 h); 2011: 50 members (full CONUS to 36 h); 2012: 28 members; 2013: 27 members at 00Z (to 48 hours) and 8 members at 12Z (to 18 hours) • Primarily examine explicit storm attributes, especially hourly maximum fields (HMFs): Updraft Helicity, Updraft Speed, 10-m AGL Wind Speed, 1-km AGL Reflectivity • Ensemble display approaches include spaghetti plots, ensemble max, and neighborhood probabilities (more on next slide) • SPC began processing deterministic high-resolution runs from EMC and NSSL as the Storm-Scale Ensemble of Opportunity (SSEO) in 2011 • The 4-km AFWA ensemble was made available to SPC in 2012

4. Convection-Allowing EnsemblesNeighborhood Probabilities • Traditional ensemble probabilities of HMFs from high-resolution models are not especially useful, owing to poor agreement among members at the grid point of these fields. • Applying a binary neighborhood approach to a storm-scale ensemble improves the statistical results of HMFs in forecasting severe weather • ROI=20-40 km • Sigma=30 grid points • Same approach can also be applied to observations (e.g., radar reflectivity) for verification purposes 03 May 2008 (Harless 2010)

5. Convection-Allowing EnsemblesNeighborhood Probabilities HM Updraft Helicity > 25 m2s-2 SSEO 24-hr fcst valid 00Z 28 April

6. Convection-Allowing EnsemblesNeighborhood Probabilities Grid-Point Probability HM Updraft Helicity >25 m2s-2 SSEO 24-hr fcst valid 00Z 28 April

7. Convection-Allowing EnsemblesNeighborhood Probabilities 40-km Neighborhood Probability HM Updraft Helicity >25 m2s-2 SSEO 24-hr fcst valid 00Z 28 April

8. Convection-Allowing EnsemblesNeighborhood Probabilities 40-km Neighborhood Smoothed Prob HM Updraft Helicity >25 m2s-2 SSEO 24-hr fcst valid 00Z 28 April

9. Convection-Allowing EnsemblesSystem Comparison • OU/CAPS Storm-Scale Ensemble Forecast (SSEF) System • Since 2007; 36-hr forecasts from 00z; 12z runs began in 2013 • Primarily WRF-ARW; 4-km grid spacing; forecasts to 60hrs in 2014 • Multi-physics, multi-initial conditions: applies SREF perturbations to NAM ICs • Advanced physics, 3DVAR & radar data assimilation; available for HWT/SFE • SPC Storm-Scale Ensemble of Opportunity (SSEO) • Since 2011; 36-hr forecasts at 00z & 12z; 7 members (2 time-lagged) • Multi-model (ARW, WRF-NMM & NMM-B), multi-physics; ~4-km grid spacing • Uses available deterministic models at SPC to process as an ensemble • Basic data assimilation through NDAS; available year-round in SPC • Air Force Weather Agency (AFWA) Ensemble • Since 2012; 60-hr forecasts at 00z & 12z; 10 members; 4-km grid spacing • Single model (WRF-ARW), multi-physics, multi-initial conditions • Cold start from downscaled global model forecasts (GFS, UM, CMC) • No data assimilation; available year-round in SPC

10. Convection-Allowing EnsemblesSFE2011 Results • The Fractions Skill Score (FSS) was calculated for neighborhood probability (ROI=40 km; σ=40 km) of updraft helicity ≥ 25 m2s-2 for the SSEO/SSEF versus practically perfect hindcasts of preliminary severe weather reports (ROI=40 km; σ=120 km) during SE2011 • The SSEO had higher fractions skill score (FSS) for neighborhood probabilities of UH ≥25 m2/s2 during SFE2011 than the SSEF • The number of members included in the SSEF did not seem to have a strong impact on the statistical results for neighborhood probabilities of UH during SE2011 when verified against severe weather reports FSS 3-h periods SFE2011 3-hr [NPRS]:UH ≥25 m2s-2 valid 06Z on 02 June 2011 w/ verifying reports and practically perfect hindcast NprobUH ≥25 m2/s2 SSEO FSS = 0.84 SSEF – 24 mem FSS = 0.68

11. Convection-Allowing EnsemblesSFE2011 Results • Even for 6-h QPF, the SSEO received the highest subjective ratings relative to other operational and experimental models and ensembles during SFE2011 • Statistically, the probabilistic QPF forecasts (>0.5”) from the SSEO were typically as good as (if not better than) the SSEF during SFE2011 at various lead times SSEO favored over CAPS ensemble from Tara Jensen, DTC from Dave Novak, WPC

12. Convection-Allowing EnsemblesSFE2012 Results • During SFE2012, the SSEO outperformed the SSEF and AFWA in terms of FSS for neighborhood probabilities of reflectivity ≥40 dBZ (bug later found in SSEF) • Subjective ratings by the SFE2012 participants of HMF ensemble forecasts tended to favor the SSEO forecasts of UH over the SSEF and AFWA forecasts Hourly FSS NprobRefl ≥40 dBZ 3-hr ensemble forecast ratings (max, nprob) of UH

13. Convection-Allowing EnsemblesSFE2012 Results • The quality of the AFWA forecasts was less consistent than the SSEO forecasts • Some UH forecasts from the AFWA ensemble were very good (bottom left) while others were poor (bottom right)

14. Convection-Allowing EnsemblesSFE2013 Results • The impact of radar data assimilation in the CAPS SSEF was evident in the first 4 hours of the 12 UTC-initialized forecast. • Otherwise, there was little statistical difference in the FSS among the 00 and 12 UTC SSEO and SSEF. • Subjective ratings of 00Z ensemble HMFs were again favorable for the SSEO during SFE2013 3-hr ensemble HMF forecast ratings (max, nprob) Hourly FSS NprobRefl ≥40 dBZ

15. Convection-Allowing EnsemblesConfiguration • Why is a “poor man’s ensemble” (i.e., SSEO) performing as well as formally designed ensembles? Let’s consider some aspects of configuration for convection-allowing ensembles • Single model vs. multi-model • Number of members • Initial conditions and IC/LBC perturbations • Physics

16. Convection-Allowing EnsemblesConfiguration: Single model vs. multi-model • Even with the same initial conditions, clustering often occurs by model core • Generally more confident in a solution if different model cores are in agreement • Is a multi-model approach a good way to address uncertainty in a convection-allowing ensemble? SSEO WRF-ARW WRF-NMM 21Z on 16 April 2011 3-hr spaghetti plot of UH ≥25 m2s2

17. Convection-Allowing EnsemblesConfiguration: Single model vs. multi-model • Is the success of the SSEO a fortuitous balance of members with an underforecast bias and those with an overforecast bias (Row and Correia, 2014 AMS); not necessarily a result of using multiple model cores? • Neighborhood verification of radar reflectivity reveals members with lower biases (e.g., NAM Nest) versus those with higher biases (e.g., NSSL-WRF) • Biases will change with upcoming HiResWupgrade, so we may learn more this spring

18. Convection-Allowing EnsemblesConfiguration: Number of members • For the way convection-allowing ensembles are currently configured, there does not appear to be a huge benefit to running more than ~10 members • Clark et al. (2011) objectively identified the “point of diminishing returns” at 9 members for 6-hr QPF at f30 and 2-km scale from Clark et al. (2011) • Could run additional members to more effectively sample the forecast PDF, but is it worth the additional computational cost? Use a larger neighborhood?

19. Convection-Allowing EnsemblesConfiguration: Initial conditions and IC/LBC perturbations • Currently, all members of the SSEO are initialized from the NAM (including two time-lagged members), so diversity primarily arises from multi-model/physics • AFWA approach (single model, 3 different IC/LBCs) often leads to higher, overconfident probabilities • SSEF approach not an obvious improvement over single, unperturbed IC (i.e. SSEO), suggesting ICs not properly perturbed at this scale AFWA OBS high probs nothing observed

20. Convection-Allowing EnsemblesConfiguration: Initial conditions and IC/LBC perturbations • In four test runs for May 2013, Kong et al. (2014) found larger domain-average ensemble spread for multiple fields by directly using LBCs from SREF members rather than extracting perturbations and applying to the NAM (current strategy) • NSSL-WRF ensemble this spring will directly utilize IC/LBCs from selected SREF members from Kong et al. (2014) Ensemble Spread

21. Convection-Allowing EnsemblesConfiguration: Physics diversity • Though spread in an ensemble with physics-only diversity is less than that from an ensemble that also includes IC/LBC perturbations, the contribution to spread from physics diversity can be large, including for instability fields (Clark et al. 2010) from Clark et al. (2010)

22. Convection-Allowing EnsemblesSummary • Convection-allowing ensembles (~4-km grid spacing) can provide useful information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. in generating outlooks on Day 1; setting the stage for the continuous flow of probabilistic hazard information down toward the warning scale • The SPC SSEO has proven to be as useful/skillful as formally designed convection-allowing ensembles, which raises questions about effective/proper configuration of these types of systems • NSSL is running eight 00Z members this spring with only IC/LBC diversity directly from 21Z SREF members • CAPS is planning to run an experimental 4-km EnKF SSEF system this year in near real-time for comparison with traditional SSEF forecasts