It Never Rains But It Pours: Modeling Mixed Discrete-Continuous Weather Phenomena

1. It Never Rains But It Pours:Modeling Mixed Discrete-Continuous Weather Phenomena J. McLean Sloughter Based on research being conducted under Adrian E. Raftery and Tilmann Gneiting This work was supported by the DoD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745.

2. Ensemble Forecasting • Single forecast model is run multiple times with different initial conditions • Ensemble mean tends to outperform individual members • Spread-skill relationship: spread of forecasts tends to be correlated with magnitude of error • Model is underdispersive (not calibrated)

3. Spread/Skill Plot Spread = max forecast – min forecast Skill = abs(forecast mean – observed)

4. Ensemble Member Forecasts 48-hour forecasts for precipitation at 5pm Oct 20, 2003 From http://www.atmos.washington.edu/~emm5rt/ensemble.cgi

5. Bayesian Model Averaging • Weighted average of multiple models • Weights determined by posterior probabilities of models • Posterior probabilities given by how well each member fits the training data • Weights, then, give an indication of the relative usefulness of ensemble members

6. BMA for ensembles where is the forecast from member i, is the weight associated with member i, and is the estimated distribution function for Y given member i Picture taken from Raftery, Balabdaoui, Gneiting, and Polakowski (2003), “Calibrated MesoscaleShort-Range Ensemble Forecasting Using Bayesian Model Averaging.”

7. The Trouble With Our Models • Forecasts never predict zero – artifact of differential equations used to create forecasts • Observed wind speed is often zero • Wind speed, even ignoring zeroes, is not normally distributed

8. What Wind Speed Looks Like Wind Speed Histogram – Several exceptionally high values make it harder to see clearly Wind Speed Histogram truncated to only go up to fifty – there is a spike at zero Wind Speed Histogram without zeroes

9. What Forecasts Look Like Forecast histogram on left (all eight forecasts have similar histograms) and observed histogram on right – even after removing zeroes from the actual histogram, the shape is still not quite right – actual is more sharply skewed.

10. The Problem With Reality • As we saw, the histograms for forecasts do not match the histogram for observations very well • Maximal observed value is 124.000 • Maximal values for each model:

11. More Trouble With Reality Pairwise correlations – Y is observed value, others are the various forecasts

13. Time Trends Left - average observed wind speed per day Right – same, but smoothed to average over 3-week interval

14. Time Trend Troubles • Higher winds in summer, lower in winter (note that this appears to be an odd trait of the northwest) • Need model to reflect seasonal patterns • Would still like to just have a simple model based on forecasts

15. A Recap Of All The Things That Make Life Interesting and Miserable • Distribution not normal • Time • Forecasts not very highly corellated with observations • Zeroes

16. What to do about distributions? • Model using another distribution – Gammas and Weibulls are popular models for windspeed • Can apply a transformation to the data Left: Root of forecast windspeed from model 1 Right: Root of observed windspeed (excluding zeroes for easier visualization)

17. What to do about time? • Rather than using all available data as training set, only train on recent data • Trade-off between lower variance of estimates with more data, and better picture of current trends with less data • Previous research (on temperature and pressure) indicates 40-day window of training data seems to give a good balance

18. What to do about the forecasts? • We’re not making the forecasts, just using them • We can apply a bias-correction by performing a linear regression of observed on forecasts (this is commonly done in forecasting already) • We can see from our weight terms which models perform better and which perform worse, and report that to the folks making the forecasts • We can hope that the science of meteorology continues to move forward as it has thus far

19. What to do about zeroes? • We need a model that includes a point mass at zero • Two main possibilities: • We could model a weighted average of eight distributions, each of which is a normal plus a point mass at zero • Or, we could first model probability of zero or non-zero, then, conditioned on non-zero, the weighted average of eight normals • We will pursue the second option for now

20. So, let’s get to it then • Probability of zero can be modeled by a logistic regression on the eight forecasts • Then, the weighted average of normals can be determined by the EM algorithm • Assume each normal has the bias-corrected forecast as its mean, and has a constant variance • Alternate between predicting membership based on weights and variances, and weights and variances based on membership • Make sure to also include probability of being non-zero when evaluating our functions

21. Let’s try out a simple test case first • Generate a sample of 100,000 ordered triples (x1, x2, y), x1 uniform over 30 to 50, x2 uniform over 10 to 20 • logistic regression coefficients of a=10, b1=-.2, b2=-.6 • with probability determined by logistic regression, y=0 • otherwise, with probability .6, y is normal around x1 with sd of 1, and with probability .4 is normal around x2 with sd of 3.14

22. How did we do? • Predicted logisitic coefficients of a=10.257, b1=-0.206, b2=-0.600 • weights of 0.598 and 0.402 • sds of 1.000 and 3.414 • Seems to be able to model pretty well under ideal artificial conditions • So now let’s try the real thing

23. How do we do? RMSE from creating forecasts for 33 days, using 40 day training periods – black is without modeling zeroes, red is with modeling zeroes

24. Iterations to convergence again, black is without modeling zeroes, red is with

25. How do we feel about this? • Including modeling of zeroes doesn’t appear to help our error much (p=0.4734), which is somewhat disappointing • However, we get our model much faster (p=2.104e-08), which is a concern when having to do a lot of these

26. What We Would Have Done Next Had We Not Been Distracted By More Pressing Matters • Consider fitting different distributions rather than using a transformation • Fit a model with point masses at zero for each individual component • Try additional bias corrections • Compare results for different training windows • Investigate importance of starting values in EM algorithm • Evaluate performances of prediction intervals rather than just prediction means

27. Three Months Later… • In the distraction interim, precipitation data became available • Precipitation forecasts tend to be better than wind forecasts • Weather people tend to be more interested in precipitation forecasts • And so, we have abandoned (for now) wind speed, and are looking at precipitation instead

28. How is rain different? • Distribution doesn’t look normal, even under transformations • Models do predict zero, but we still see point masses at zero

29. Conditional Histograms Observed given forecast from 1.5 to 4 Observed given forecast from 55 to 80 in .01”

30. Fitting Gammas Shape Parameter Rate Parameter Coef. Of Variation

31. Something’s Not Quite Right Sample Mean Estimated Mean

32. At least something worked out nicely Proportion of Zeroes

33. And Now? • First, find out what’s going funny with our gamma fitting • Then, try to come up with a way to do some sort of gamma fitting in the EM algorithm • Then, look at all those things we wanted to look at before