Introduction to EKDMOS

1. Introduction to EKDMOS Matt Peroutka Meteorological Development Lab National Weather Service, NOAA Bob Glahn Jerry Wiedenfeld John Wagner Greg Zylstra

2. Reference • Glahn B., M. Peroutka, J. Wiedenfeld,J. Wagner, G. Zylstra, et al. (2008) MOS uncer-tainty estimates in an ensemble framework. Monthly Weather Review: In Press. • Available online at Monthly Weather Review Early Online Releases. • AMS estimates publication in December 2008.

3. Introduction—1 • To date, MDL's MOS guidance has used multiple regression to produce single-valued forecasts for • Temperature (T) • Dew point (Td) • Daytime maximum temperature (MaxT) • Nighttime minimum temperature (MinT) • Two changes • Ensembles now available • Emerging requirement for probabilistic forecasts of these four weather elements

4. Introduction—2 • Regression yields error estimate from which probabilities can be calculated. • Probabilistic forecasts from a single model run • Distribution of weather element must be near normal • Why haven't we used this before now? • Not needed to get good single-valued forecasts. • No big push for probabilistic guidance. • We now have ensemble runs, and we sought a way to combine ensembles and regression in some way.

5. EKDMOS Method • Use screening multiple regression with the NCEP's GEFS ensemble means, • Estimate error variance directly from the regression, • Apply equations to individual ensemble members and combine results with Kernel Density fitting • Gaussian kernel • Standard deviation produced by the regression • Apply a spread adjustment based on the spread of the ensemble members.

6. EKDMOS Method Development Error Variance Ensemble Means Regression Observations Predictive Equation Implementation Forecast Spread Adjustment Kernel Density Fitting Ensemble Members Probability Distribution

7. Error Estimation in Linear Regression • The linear regression theory used to produce MOS guidance forecasts includes error estimation. • The Confidence Interval quantifies uncertainty in the position of the regression line. • The Prediction Interval quantifies uncertainty in predictions made using the regression line. The prediction interval can be used to estimate uncertainty each time a MOS equation is used to make a forecast.

8. Estimated Variance of a Single New Independent Value • Estimated variance • Where

9. Computing the Prediction Interval The prediction bounds for a new prediction is where t(1-α/2;n-2) is the t distribution n-2 degrees of freedom at the 1-α (two-tailed) level of significance

10. Multiple Regression (3-predictor case) 3-predictor Matrix Predictand Vector Coefficient Vector

11. Multiple Regression, Continued Error bounds can be put around the new value of Y with where • s2 is the variance of the predictand, • R2 is the reduction of variance, • X’ is the matrix transpose of X, and • ()-1 indicates the matrix inverse.

12. Regression Equation and Prediction Intervals

13. Kernel Density Fitting Forecasts are combined with Kernel Density Fitting, using a Gaussian kernel and the standard deviation produced by the regression.

14. EKDMOS Spread Adjustment x = ratio of new width to old width

15. Applying the Spread Adjustment • Method • KDE used to compute unadjusted distribution. • Average of forecasts used as center of distribution. • Each point on the distribution adjusted to be closer to center of distribution by width ratio (x). • Spread factor (sf) empirically determined. • Values of 0.4 and 0.5 have worked best. • Same value for both seasons and all weather elements.

16. EKDMOS Comments (T, Td, MaxT, MinT) • Using ensemble means as predictors seems very effective. • Prediction interval from regression can be used alone to make reliable forecasts. • Distributions will be symmetric. • Combination of prediction interval and ensembles can yield skewed distributions.

17. QPF is not Quasi-Normal! • One-predictor equations, prediction intervals, and data for QPF (top) and cube root QPF (bottom). • Regression line fits poorly. Data points well beyond limits of prediction interval. • Cube root improves the fit, but makes more problems. Note artificial "floor" at 0.22 in—cube root of 0.01 in. • We need a different approach...

18. Method for EKDMOS 6-h PQPF • Use screening multiple regression with the NCEP's GEFS ensemble means to forecast QPF ≥ 0.01 in (PoP06). • Use same regression technique to forecast conditional QPF for other categories. • Apply equations for each category to individual ensemble members. • Multiply the PoP06 by each categorical probability to create unconditional probabilities for each category and member. • Ensure probabilities are monotonically increasing and between 0 and 1. • Combine ensemble results for each category with arithmetic mean. Combination of EKDMOS temperature technique and GFS MOS QPF technique.

19. EKDMOS Comments (QPF) • Using ensemble means as predictors seems very effective. • Probability distributions can be generated from a single model run. • Combination of regression and ensemble modeling seems to yield better forecasts at long lead times.

20. Evaluation • Tools • Probability Integral Transform (PIT) Histograms • Square Bias (SB) • Cumulative Reliability Diagram (CRD) • Continuous Ranked Probability Score (CRPS) • Baseline for Comparison • Raw Ensembles • Operational Ensemble MOS

21. Probability Integral Transform (PIT) Histogram • Graphically assesses reliability for a set of probabilistic forecasts. Visually similar to Ranked Histogram. • Method • For each forecast-observation pair, probability associated with observed event is computed. • Frequency of occurrence for each probability is recorded in histogram as a ratio. • Histogram boundaries set to forecast percentiles. Ratio of 1.795 indicates ~9% of the observations fell into this category, rather than the desired 5%. Ratio of .809 indicates ~8% of the observations fell into this category, rather than the desired 10%. T=34F; p=.663 0.1 0.2 0.95

22. Probability Integral Transform (PIT) Histogram, Continued • Assessment • Flat histogram at unity indicates reliable, unbiased forecasts. • U-shaped histogram indicates under-dispersion in the forecasts. • O-shaped histogram indicates over-dispersion. • Higher values in higher percentages indicate a bias toward lower forecast values.

23. Squared Bias in Relative Frequency • Weighted average of squared differences between actual height and unity for all histogram bars. • Zero is ideal. • Summarizes histogram with one value. Sq Bias in RF = 0.057

24. Cumulative Reliability Diagram (CRD) • Graphically assesses reliability for a set of probabilistic forecasts. Visually similar to reliability diagrams for event-based probability forecasts. • Method • For each forecast-observation pair, probability associated with observed event is computed. • Cumulative distribution of verifying probabilities is plotted against the cumulative distribution of forecasts. 63.5% of the observations occurred when forecast probability was 70% for that temperature or colder. 0.7

25. Evaluation • Tools • Probability Integral Transform (PIT) Histograms • Square Bias (SB) • Cumulative Reliability Diagram (CRD) • Continuous Ranked Probability Score (CRPS) • Baseline for Comparison • Raw Ensembles • Operational Ensemble MOS

26. Baseline • Rank-ordered forecasts can be used to form a crude CDF. Weibull plotting position estimator: • Technique applied to raw ensemble output (RawEns) and Operational Ensemble MOS (ENSMOS). ENSMOS as seen on http://eyewall.met.psu.edu

27. Baseline Reliability OpnlEnsMOS OpnlEnsMOS Raw Ens Raw Ens Day2 Day5 Day7

28. EKDMOS: Dep. and Ind. Data Dep Ind Dep Ind Day2 Day5 Day7

29. EKDMOS: Ind. Dew Point, Daytime MaxT, Nighttime MinT Dew Point MaxT MinT Day2 Day5 Day7

30. Continuous Ranked Probability Score The formula for CRPS is where P(x) and Pa(x) are both CDFs and

31. Continuous Ranked Probability Score • Proper score that measures the accuracy of a set of probabilistic forecasts. • Squared differ-ence between the forecast CDFand a perfect single value forecast, inte-grated over all possible values of the variable. Units are those of the variable. • Zero indicates perfect accuracy. No upper bound.

32. Reliability/Accuracy

33. Reliability/Accuracy

34. Reliability/Accuracy

35. Accuracy for Td, MinT, MaxT Td MinT MaxT

36. 6-h QPF Reliability at 24-h ≤0.01 ≤0.10 ≤0.25 ≤0.50

37. 6-h QPF Reliability at 72-h ≤0.01 ≤0.10 ≤0.25 ≤0.50

38. 6-h QPF Reliability at 120-h ≤0.01 ≤0.10 ≤0.25 ≤0.50

39. 6-h QPF Accuracy

40. Disseminating EKDMOS • EKDMOS distributions are frequently skewed and occasionally bimodal. • Precludes fitting analytical distributions. • Rather, disseminate points on the CDF—percentile temperatures. • 13 probability levels (0.05, 0.10, 0.20, 0.25, 0.30, 0.40, 0.50, 0.60, 0.70, 0.75, 0.80, 0.90, 0.95) • Allows users/partners to choose a threshold temperature and compute an exceedance probability. • Single-value forecasts will be created as well. • NDGD grids. • Tentative time projections: • Days 1-8 x 3-h (T/Td). • Days 1-16 x 24-h (MaxT/MinT).

41. Chance of temperature below40.0 degrees F is 67.9%. Median of the distribution 38.3 degrees F. 50% Confidence Interval (35.8, 40.7) degrees F. 20% chance of temperature below 35.2 degrees F. 90% Confidence Interval (32.2,44.3) degrees F. 72-h T Fcst KBWI 12/14/2004

43. PDFs for a Single Event (PAJN)

44. EKDMOS Products/Services • Grids in NDGD • Partners might find most value in EKDMOS. • AWIPS • XML via SOAP web service (BAMS, April 2008) • Popular outlet (millions of queries/day) for NDFD grids today. • Will support EKDMOS queries as well. • Will support queries for user-defined thresholds. • Meteograms/Plume Diagrams • Limited number of graphics

45. Meteograms • Uses 3 probability levels (0.10, 0.50, 0.90) • Time series of forecast values • Mullin Pass, Idaho (KMLP)

46. "Wings of Uncertainty"

47. Conclusions • Developed method for making MOS forecasts from ensembles for quasi-normally distributed variables. • Regression applied to NCEP ensemble means. • KDE, normal kernel, mean and standard deviation from regression. • Spread Adjustment. • Reliable and Robust. • Will be implemented in NDGD.

48. Introducing EKDMOS