ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

1. ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE Jeff Waldstreicher Scientific Services Division – Eastern Region Northeast Regional Operational Workshop (NROW) November 4-5, 2003 Albany, NY

2. ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS • IMPACT ON VERIFICATION SCORES • Performance Metrics • CASE STUDY ANALYSIS • Event Verification • Subjective Evaluation of Impact on Forecast Process • AFDs • Event Reviews and Impact Reports

3. COMPLICATIONS • Many Factors Influence Performance Metrics • Factors Not Independent • Difficult to tie performance changes to a specific factor • Cannot Analyze “Null Case” • If forecasters have “knowledge” or “data,” cannot directly answer “What if they did not have the knowledge?”

4. FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES • Infusion of New Technologies • Hardware (New systems or Processors) • Software (New algorithms or Models) • Applied Research and Development • National (Research Laboratory) • Local/Regional • Independent • Collaborative • Changes to Operational Procedures • Implementation of Best Practices

5. FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES • Climate Variability • Variations in frequency and type of events • External Outreach and Education • Including development of spotter networks • Personnel/Staffing Issues • Including forecaster experience levels • Training • All other factors are also tied to training issues

6. HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS? • Can be global (national), regional, or local • Technology usually national • Climate Variability typically regional • Staffing usually local • Impact can be long or short term • Technology infusion typically has long term impact • Climate Variability is usually short term • Factors are often inter-related • New technology or staffing changes require training • Applied R&D often yields/suggests changes to operations

7. COMET PROJECTS IN EASTERN REGION • Different types of projects • Cooperative (2-3 year ~$35K/yr) • 19 ER Cooperative Projects funded since 1991 • Partners (1 year ~$9K/yr) • 39 ER Partners Projects funded since 1991 • 20 ER WFOs and 3 RFC have participated • 90 Offices Nationwide • 21 Universities have participated in ER projects • 70+ Universities Nationwide

8. EVALUATING COMET PROJECTS • Examined COMET Cooperative and Partners Projects in Eastern Region • Projects completed between 1995 and mid-2001 • Projects specifically addressing warning programs • Tornado • Severe Thunderstorms • Flash Flooding • Winter Storms • Study was designed to minimize as much as possible the impact of the factors previously discussed.

9. METHODOLOGY • 3-year running verification scores used • Minimize impact of short-term factors such as variability of events • Compared 3 years before project to 3 years following project • 1996-mid 2001 period helped ensure 3-years of post-88D data in “before” scores and a full 3 years of “after” scores • 3-year “expected” improvements calculated based on long term trend of ER-wide scores • Compared rate of improvement for WFO involved in collaborative project to ER-wide improvement • Use of ER scores as a baseline minimizes the impact of national/region-wide factors such as AWIPS and radar system improvements • Very difficult to evaluate the impact of project results beyond the primary WFO

16. INDIVIDUAL PROJECT COMPARISON • Compare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period

17. PROBABILITY OF DETECTION (POD) # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 3 2 1 Severe Tstm Warnings 6 1 0 Flash Flood Warnings 5 1 0 Winter Storm Warnings 4 3 0

18. FALSE ALARM RATIO (FAR) # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 2 4 0 Severe Tstm Warnings 3 4 0 Flash Flood Warnings 3 3 0 Winter Storm Warnings 3 3 1

19. LEAD TIME # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 5 1 0 Severe Tstm Warnings 5 1 1 Flash Flood Warnings 4 1 1 Winter Storm Warnings 4 3 0

20. IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES • WFO RAH and NCSU • Continuous collaborative projects since January 1991 • COMET Projects 1991-2000 • 3 Cooperative Projects • 3 Partners Projects • 1 Graduate Fellowship • CSTAR Projects • 2000-2003 • 2003-2006 • Unique “laboratory” to examine the impact of long-term collaborative activities

27. IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES • WFO ALY and U-Albany • Continuous since February 1995 • Numerous COMET Projects • 3 Cooperative Projects • 7 Partners Projects • Several additional projects between UA and NCEP • CSTAR PROJECT 2001-2004

34. SUMMARY • Verification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance • There are indications that certain performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.