Marine Verification at WFO Key West Where to go and how to get there… Andy Devanas, SOO and

1. Marine Verification at WFO Key West • Where to go and how to get there… • Andy Devanas, SOO and • Matt Parke, Verification Focal Point

2. There are three kinds of lies: • lies, damned lies, and statistics. • Benjamin Disraeli (1804 - 1881)

3. Figure 1 USF Buoy Not Operational X X X X X X Marine Area approx. 22000 square miles Massachusetts – 10555 square miles X– No Surface Data Available NO WAVE DATA AT ALL

6. Key West Verification System Needs… Need to integrate verification into the forecast process Need to provide meaningful, frequent and timely verification data to forecasters ( e.g. Percent Correct vs. ESS) Need to identify forecast and guidance biases Enhanced discrimination in critical categories

7. How To Get There… • *Use MVerify for rapid feedback to forecasters • -Local AWIPS application • -Provides most flexibility in stratification of desired parameters • Minimum forecast period adjustable • Can segregate by forecaster • *Investigate various statistical measures/distributions which best track forecast performance • *Set reasonable goals of forecast performance. • *Monitor forecast performance. • -Accountability • *Use verification statistics in post-mortem evaluations as feedback to forecasters.

8. APPLICATION OF THE NATIONAL MARINE VERIFICATION PROGRAM (NMVP) TO LOCAL MARINE VERIFICATION • DRAWBACKS • *NMVP is not timely. Must wait until after end of month to access statistics. • *NVMP does not allow defining of periods. • Minimum period is one month. • *Can’t define inter-month periods. • *NMVP does not provide access to individual forecast-observation pairs. • -Cannot do post-mortem analysis. • -Do not know which matched pairs are producing statistics. • -No Quality control of forecasts and observations. • *Some measures (ESS) are heavily weighted towards rare climatological events. • -ESS is volatile. • -Critical missing matched pairs can have large impact on ESS. • BENEFITS • *Provides relatively easy and quick statistics spanning > 3 year dataset • *Can stratify data with regards to forecast period, buoy (cman), and season

9. Equitable Skill Score reward/penalty Characteristics (from NMVP)- • A relatively small reward for correctly forecasting common events • A large reward for correctly forecasting rare events • Less penalty is assigned to an incorrect forecast of a rare event than a similar size error of a common event. "Near hits" of rare event receive a modest reward. • Uses a graduated reward/penalty system, whereby a large forecast error is penalized more than a small forecast error for a given category of events. What’s wrong with this picture??

10. ESS Does Not Account For Value of Forecast to Our Customers… MOST CRAFT REMAIN IN PORT S C E C S C A

11. Before Matched Sets at NMVP Site

12. EYW Forecasts: January 2004 This Forecast Was Never Made Where Did This Number Come From? This Will Severely Skew ESS Score. As Well As Other Statistical Measures.

13. Percent Correct Definition From NOAA Verification Glossary PC= (A+D)/(A+B+C+D).

14. Percent Correct Definition From Marine Verification Program Glossary Percent Correct Definition: Percentage of the correct forecasts (hits) from the total number of events. Formula: In Other Words: A/(A+B+C+D)

15. “gaming”

16. Some EYW verification results: • The majority, almost 89%, of our observations (marginal distribution p(x)) were ≤ 17 kts. • Our forecast system had a hard time discriminating between light winds (< 8kts) and winds of 8 – 12 kts. • Our forecast system is slightly better discriminating in the 8 – 12 kt range, but still tends to overforecast more often than underforecast. • At 13 – 17 kts, our system is the least biased. • We underforecast (bias > 1) moderate to high wind events. • Forecasters tend to “game” light wind forecasts which do not have much value to customers.

17. Gridded Verification at EYW…. Otherwise known as… What are you Kidding me??

18. From Murphy 1993… “Since Forecasters’ judgments necessarily contain an element of uncertainty, their forecasts must reflect this uncertainty accurately in order to satisfy the basic maxim of forecasting. In general, then, forecastsmust be expressed in probabilistic terms.”

19. Melbourne’s ADAS Closer to grid verification *Supported by relatively dense surface data network over land *Buoy in Marine Zones

20. LAPS analysis highly influenced by single data point. Winds in this region most likely NE 10-15 knots.

21. Again – note influence of single data point over huge region…

22. We do not have ANY guidance to forecast wave height inside the reef at this resolution and detail, and little skill – therefore we do not attempt to forecast this region in our grids and zero out this region – but do include in our CWF. Wavewatch III is poor at best outside of the reef – and does not account for gulf stream interactions. AND NOT ONE WAVE HEIGHT DATA POINT!! So How Do You Verify Wave Height?? Answer: You don’t.

23. Marine Forecast 730 PM EST SUN MAR 14 2004 SMALL CRAFT ADVISORY TONIGHT: EAST WINDS AROUND 20 KNOTS. SEAS OUTSIDE THE REEF 5 TO 7 FEET...EXCEPT HIGHER IN THE GULF STREAM. SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS CHOPPY TO ROUGH. SCATTERED SHOWERS. MONDAY: EAST WINDS 15 T0 20 KNOTS...DECREASING TO 10 TO 15 KNOTS BY NOON. SEAS OUTSIDE THE REEF 4 TO 7 FEET...EXCEPT HIGHER IN THE GULF STREAM. SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS CHOPPY. SCATTERED SHOWERS. MONDAY NIGHT: SOUTHEAST WINDS NEAR 15 KNOTS. SEAS OUTSIDE THE REEF 3 TO 5 FEET. SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS A MODERATE CHOP. ISOLATED SHOWERS.

24. GRID VERIFICATION “THOUGHTS”… • You must have relatively dense surface data network to support gridded verification field. • Perception that numerical models are the better performer are incorrect and not supported. • Need high resolution shallow water wave guidance. Given the current state of model guidance and surface data available for the EYW marine area – - It is entirely possible that our forecasters will have to DEGRADE our forecasts to match verification fields…