A STATISTICAL COMPARISON OF AMPS 10-KM AND 3.3-KM DOMAINS

1. A STATISTICAL COMPARISON OF AMPS 10-KM AND 3.3-KM DOMAINS Michael G. Duda, Kevin W. Manning, and Jordan G. Powers Mesoscale and Microscale Meteorology Division, NCAR AMPS Users’ Workshop 2004 June 8-10, 2004

2. Introduction • Purpose: • Demonstrate the usefulness of statistical significance testing in comparing biases of two domains • Determine where biases at McMurdo Station are significantly different in the 3.3-km and 10-km AMPS domains • Examine a 7 day period beginning 12Z Nov. 27, 2003 when McMurdo Station was affected by a snowstorm • Methodology: • Use hypothesis testing to identify statistically significant differences in mean bias • Consider only differences that are statistically significant

3. Domain Configuration Compaq OSF/Alpha Linux/Xeon (SPAWAR machine)

4. Forecast Analysis Times

5. Why Consider Statistical Significance? • Mean bias curves do not indicate the variance in the biases • Some differences between curves are not as relevant

6. Hypothesis Testing • Consider biases to be from a hypothetical population (assumed to be normally distributed) • Let d = x3.3 – x10 • x3.3 and x10 are biases in 3.3-km and 10-km domains at a given time • Perform one-sample Student’s t test • H0: d=0 • Reject H0 with 95% confidence if t t • Test statistic:

7. Hypothesis Testing Example Circled pressure levels will be examined in the next two slides

8. Example: 150 hPa Temperature • For this data we can reject the null hypothesis at the 5 percent level • This means we reject the hypothesis that the means of the 3.3-km and 10-km bias populations are the same differences between curves

9. Example: 850 hPa Temperature • For this data we cannot reject the null hypothesis at the 5 percent level • This means we cannot reject the hypothesis that the 3.3-km and 10-km bias populations have the same mean differences between curves

10. Comparison Results: Temperature • Statistically significant differences • Surface: 3.3-km grid has warm bias while 10-km grid has a cool bias at hours 24, 36 • 925 hPa: 3.3-km grid has warm bias while 10-km grid has a cool bias at hours 24, 36 • 300 hPa: 3.3-km grid has larger warm bias than 10-km grid • No statistically significant differences • At hours 24 and 36, no significant differences in MAE at any level

11. 24hr Temperature (Mean Bias)

12. 36hr Temperature (Mean Bias)

13. 24hr Temperature (MAE)

14. Comparison Results: Wind U-Component • Statistically significant differences • Surface: 3.3-km grid has lower positive bias than 10-km grid at forecast hours 12, 24, 36 • 850 hPa: 3.3-km grid has larger negative bias at forecast hours 12, 24, 36 • 500 hPa: 3.3-km grid has smaller bias, but MAEs of both grids are similarly large • Differences at other levels are not statistically significant

15. 24hr Wind U-Component (Mean Bias)

16. 36hr Wind U-Component (Mean Bias)

17. 24hr Wind U-Component (MAE)

18. Example: Surface Temperature 35 hr forecast valid 23Z Dec 01, 2003 10-km domain 3.3-km domain

19. Summary • Use a Student’s t test (at 5 percent level) to perform statistical significance testing on difference between 3.3-km and 10-km biases • Identify statistically significant differences on model bias v. pressure plots for McMurdo • Consider only statistically significant differences between mean biases to improve objectivity • Apparently large differences in mean bias may be statistically insignificant and misleading

20. Questions?

21. Hypothesis Testing Example * Biases at these pressure levels will be examined in the following slides * *

22. Example: 400 hPa Wind V-Component For this data we do not reject the null hypothesis at the 95 percent level differences between curves

23. Example: 925 hPa Wind V-Component For this data we do reject the null hypothesis at the 95 percent level differences between curves