A Proposed Decision Market for Air Quality Forecasting

1. A Proposed Decision Market for Air Quality Forecasting William F. Ryan Andrew N. Kleit Brett F. Taubman Department of Meteorology The Pennsylvania State University EPA 2006 National Air Quality Conference San Antonio, Texas

2. What is a Prediction Market? • A prediction market is a market run for the primary purpose of using the information content provided by market values to make predictions about specific future events. • In a decision market you “invest” in whether or not a certain event is going to happen.

3. Also Known As….. • Decision markets • Event-driven derivative markets • Information aggregation markets • Decision support markets • Pseudo-markets • And many others……….

4. Why Are Prediction Markets Useful? • They work! Predictions based on information provided by markets can out-perform alternatives. • Iowa Electronic Market (IEM): Over last 4 Presidential elections, when compare market estimate from IEM to 600 polls (on day of poll), the IEM was closer to the actual vote total in 75% of cases (final error of 1.4%) • Hewlett-Packard Employee Sales Forecast - Beats company official forecast in 75% of cases.

5. Why Do Market Predictions Work (1)? • Markets serve to aggregate information • Well known in weather forecasting circles that consensus (many forecaster) forecasts out-perform any single forecaster • See, Sanders, F., 1973: Skill in forecasting daily temperature and precipitation: Some experimental results, Bulletin Amer. Meteor. Soc., 54, 1171-1179.

6. Why Are Consensus Forecasts Better? • Cumulative knowledge of all forecasters is greater than a single forecaster • As system becomes more complex, no single forecaster can reliably comprehend or integrate all relevant factors. • Mathematically, if > 2 sources of imperfect forecasts have independent and non zero skill, there exists some optimal blend of predictions that will be better than any single forecast.

7. Application to Air Quality PSU METEO 497A Mix of chemistry and meteorology students with no air quality forecasting experience. 18 forecasts (2 x weekly) of PHL PM2.5. Consensus Skill ~ Expert Median Absolute Error (μg/m3) Consensus Forecast Me

8. Consensus Principles Apply to Numerical Forecast Models As Well • There is evidence that multi-model consensus is superior to any individual control run. • Result is not just due to cancellation of overall bias but to a difference in errors with respect to each traveling disturbance that affects the weather. • See, Fritsch, J.M., J. Ross and R. L. Vislocky, 2000, Model consensus, Weather and Forecasting, 15, 571-582.

9. Why Do Prediction Markets Work (2)? • Continuously updated dynamic forecasts adjust and reflect changes in information over time. • Compare: d(prog)/dt, forecasters rule of thumb, given a set of lagged forecasts, if forecasts show a trend, this trend is more likely than not to continue and provide information to correct most recent forecast. • Similarly, if poor continuity in forecast models, can often assume increase in uncertainty in forecast. • But, what is the extent of uncertainty?

10. Why Do Prediction Markets Work (3)? • As applied in the form of decision markets, consensus forecasts are particularly helpful in providing probability or spread in forecasts. • Always want to know the confidence interval around a mean estimate. If can create some ~ normal distribution then can determine ± 2σ (95% confidence interval).

11. Application to Air Quality • Currently, air quality forecasts are deterministic – guided by statistical or numerical model output - and issued by a small group (or sometimes just one) forecaster. • Forecasts are usually given as a color code. • Is forecast “high” or “low” Orange? What is the real threat of Code Red? • Even ppbv forecasts are issued without information on the spread of probablility.

12. Why is Uncertainty Information Useful?: Example from the Electricity Market • Electricity supply is finite but typically > demand. • So price (supply curve at left) of energy is pretty flat until get close to maximum capacity and then it increases rapidly (“hockey stick”) Supply Curve Price Capacity

13. Example: Electricity Market • In summer, excess electricity demand is driven by consumer air conditioning (a/c) use. • a/c use is determined by temperature. • Roughly, if know temperature, know demand. • If σ of temperature forecast small, see left, little impact on expected price. T T - 2σ T + 2σ Price Capacity (Demand)

14. Example: Electricity Market • What if σ is much larger in a given case? • Here, important effect on the price of electricity possible. • If you bought/sold electricity, you would like to know this, right? T T - 2σ T + 2σ Price Capacity (Demand)

15. How Do Decision Markets Operate? • Real money market • Floats a prediction • An event-driven futures market. • Price range is [0,1] where price is understood as a probability. • Normal distribution is found that best fits distribution of market prices to provide measure of uncertainty (mean, σ).

16. Example: Simple Trade • I offer to pay you $1 if Code Red O3 is observed tomorrow in Philadelphia. • What will you pay me in return for this promise? Or, What price will I place on this offer? • If forecast is 95ºF for tomorrow, there is a decent chance of Code Red, I might have to pay so I will charge a high price for this promise, say 75 cents (0.75)? • If there is a change of thunderstorms? You might think Code Red a bit less likely, you might offer only 0.65. • A deal may be struck and many deals like that with the final price reflecting the probability.

17. What Are the Market Requirements? • For decision markets to operate effectively need: • Well specified future event • Enough traders to bring liquidity and generate the efficient price. • Openness with respect to information. • Diversity, independence and de-centralization • No single source dictates decisions.

18. Other Requirements • A problem for which a probability forecast is needed and not provided. • A legal exchange • Foreign country (www.tradesports.com) • Unique regulatory authority, e.g., IEM has waiver from the CFTC. • Self funded: Traders are given a “stake”.

19. Do Decision Markets Always Work? • Markets that work well • H-P Internal Sales Market • Iowa Influenza Market • Variety of political markets • Markets that didn’t • MAHEM (hurricane market) • Already get probabilities from the NHC • Market not routine, daily event.

20. Test Market • Penn State (Spring, 2006) three classes (ENNEC 473 and METEO 497A (Air Quality Forecasting) and Smeal Business School) – mix of expert and non-expert. • Forecast Philadelphia maximum temperature out to five days. • Privately funded market.

21. PSU Temperature Test Market

22. Proposal: Ozone Forecast Market • Problem in Philadelphia: When forecast Code Red, almost always observe Code Red. Few false alarms. • But, when observe Code Red, forecasts are typically split between Code Orange and Code Red forecasts. • Typically forecasters know whether or not it will be “low” Orange or “high” Orange but how quantify the probabilities?

23. Ozone Market Framework • Identify a series of bins, e.g., 75-85 ppbv, 85-95 ppbv, etc…. • Traders then sell contracts within each bin. • e.g., if I think thunderstorms likely early in the day, I will sell contracts at 105-115 ppbv because I expect O3 to be lower and I will buy at lower ppbv values. • Sales are limited by maximum funds available to each trader (margin) set by size of stake. • Normal distribution (“model price”) is fit to the price of the contracts in each bin. Best fit model provides mean and σ for forecast.

24. Details, details…. • Range of O3 concentrations is large, how set trading bins day-to-day? • Who sets trading bins? Who “makes” the market? • How much lead time to trade? 24, 48 hours? More? • When close the market? At time of official forecast?

25. Is This A Good Market? • Well specified event • Maximum O3 at defined set of monitors • But, QA/QC issues related to AIRNow? • Are there enough traders? • Enough air quality forecasters? Students? Affected community? • What knowledge required for non-experts? • Is information open? • Weather forecast are open, NOAA/EPA AQF model is open but what about private forecast models?

26. What do We Need? • Hoping for Summer, 2007 launch • Money • Participants • If interested at all: email: wfr1@psu.edu • Specific definition of market.

27. Conclusions • Prediction markets use information provided by market values to make predictions about specific future events. • Prediction markets have been a skillful forecast tool in some applications. • Markets, as a consensus forecast, improve predictions, and their effective use, by aggregating information and providing a measure of certainty. • Prediction markets buy and sell event-driven futures contracts. The price of these contracts, over a range of possible outcomes, can be fit to a normal distribution and provide uncertainty estimates.