1 / 46

Prediction Markets and the Wisdom of Crowds

Prediction Markets and the Wisdom of Crowds. David Pennock Yahoo! Research NYC. Outline. Prediction Markets Survey What is a prediction market? Examples Some research findings The Wisdom of Crowds: A Story. Bet = Credible Opinion. Hillary Clinton will win the election.

sunila
Download Presentation

Prediction Markets and the Wisdom of Crowds

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Prediction Markets and the Wisdom of Crowds David Pennock Yahoo! Research NYC

  2. Outline • Prediction Markets Survey • What is a prediction market? • Examples • Some research findings • The Wisdom of Crowds:A Story

  3. Bet = Credible Opinion Hillary Clinton will win the election • Which is more believable?More Informative? • Betting intermediaries • Las Vegas, Wall Street, Betfair, Intrade,... • Prices: stable consensus of a large number of quantitative, credible opinions • Excellent empirical track record “I bet $100 Hillary will win at 1 to 2 odds”

  4. Example: March Madness

  5. More Socially Redeemable Example http://intrade.com Screen capture 2007/05/18

  6. A Prediction Market • Take a random variable, e.g. • Turn it into a financial instrument payoff = realized value of variable Bird Flu Outbreak US 2007?(Y/N) I am entitled to: Bird FluUS ’07 Bird FluUS ’07 $1 if $0 if

  7. Why? Get information • price  probability of uncertain event(in theory, in the lab, in the field, ...more later) • Is there some future event you’d like to forecast?A prediction market can probably help

  8.  6 = 6 ? = 6 I am entitled to: $1 if $0 if A Prediction Market • Take a random variable, e.g. • Turn it into a financial instrument payoff = realized value of variable 2008 CAEarthquake? US’08Pres =Dem?

  9. Aside: Terminology • Key aspect: payout is uncertain • Called variously: asset, security, contingent claim, derivative (future, option), stock, prediction market, information market, gamble, bet, wager, lottery • Historically mixed reputation • Esp. gambling aspect • A time when options were frowned upon • But when regulated serve important social roles...

  10.  6 = 6 I am entitled to: $1 if $0 if Getting Information • Non-market approach: ask an expert • How much would you pay for this? • A: $5/36  $0.1389 • caveat: expert is knowledgeable • caveat: expert is truthful • caveat: expert is risk neutral, or ~ RN for $1 • caveat: expert has no significant outside stakes

  11.  6 = 6 = 6 Getting Information • Non-market approach:pay an expert • Ask the expert for his report r of the probabilityP( ) • Offer to pay the expert • $100 + log r if • $100 + log (1-r) if • It so happens that the expert maximizes expected profit by reporting r truthfully • caveat: expert is knowledgeable • caveat: expert is truthful • caveat: expert is risk neutral, or ~ RN • caveat: expert has no significant outside stakes “logarithmic scoring rule”, a “proper” scoring rule

  12. I am entitled to:  6 = 6 = 6 $1 if $0 if Getting Information • Market approach: “ask” the public—experts & non-experts alike—by opening a market: • Let any person i submit a bid order: an offer to buy qi units at price pi • Let any person j submit an ask order: an offer to sell qj units at price pj(if you sell 1 unit, you agree to pay $1 if ) • Match up agreeable trades (many poss. mechs...)

  13. Opinion poll Sampling No incentive to be truthful Equally weighted information Hard to be real-time Ask Experts Identifying experts can be hard Incentives Combining opinions can be difficult Prediction Markets Self-selection Monetary incentive and more Money-weighted information Real-time Self-organizing Non-Market Alternatives vs. Markets

  14. Machine learning/Statistics Historical data Past and future are related Hard to incorporate recent new information Prediction Markets No need for data No assumption on past and future Immediately incorporate new information Non-Market Alternatives vs. Markets

  15. $1 if $0 otherwise Function of Markets 2: Risk Management • If is bad for me, I buy a bunch of • If my house is struck by lightening, I am compensated.

  16. Risk Management Examples • Insurance • I buy car insurance to hedge the risk of accident • Futures • Farmers sell soybean futures to hedge the risk of price drop • Options • Investors buy options to hedge the risk of stock price changes

  17. Financial Markets vs. Prediction Markets

  18. What you can say/learn% chance that Hillary wins GOP wins Texas YHOO stock > 30 Duke wins tourney Oil prices fall Heat index rises Hurricane hits Florida Rains at place/time Where IEM, Intrade.com Intrade.com Stock options market Las Vegas, Betfair Futures market Weather derivatives Insurance company Weatherbill.com Giving/Getting Information

  19. http://intrade.com http://tradesports.com Screen capture 2007/05/18

  20. Intrade Election Coverage

  21. http://www.biz.uiowa.edu/iem

  22. http://www.wsex.com/ http://www.hedgestreet.com/ Screen capture 2007/05/18 Screen capture 2007/05/18

  23. Play money;Real predictions http://www.hsx.com/

  24. http://www.ideosphere.com http://us.newsfutures.com/ Cancercuredby 2010 Machine Gochampionby 2020

  25. Yahoo!/O’Reilly Tech Buzz Game http://buzz.research.yahoo.com/

  26. More Prediction Market Games • BizPredict.com • CasualObserver.net • FTPredict.com • InklingMarkets.com • ProTrade.com • StorageMarkets.com • TheSimExchange.com • TheWSX.com • Alexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ, MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds • http://www.chrisfmasse.com/3/3/markets/#Play-Money_Prediction_Markets

  27. Catalysts • Markets have long history of predictive accuracy: why catching on now as tool? • No press is bad press: Policy Analysis Market (“terror futures”) • Surowiecki's “Wisdom of Crowds” • Companies: • Google, Microsoft, Yahoo!; CrowdIQ, HSX, InklingMarkets, NewsFutures • Press: BusinessWeek, CBS News, Economist, NYTimes, Time, WSJ, ...http://us.newsfutures.com/home/articles.html

  28. Does it work? • Yes, evidence from real markets, laboratory experiments, and theory • Racetrack odds beat track experts [Figlewski 1979] • Orange Juice futures improve weather forecast [Roll 1984] • I.E.M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002] • HP market beat sales forecast 6/8 [Plott 2000] • Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’03][Schmidt 2002] • Market games work [Servan-Schreiber 2004][Pennock 2001] • Laboratory experiments confirm information aggregation[Plott 1982;1988;1997][Forsythe 1990][Chen, EC’01] • Theory: “rational expectations” [Grossman 1981][Lucas 1972] • More later …

  29. [Source: Berg, DARPA Workshop, 2002] Example: IEM 1992

  30. [Source: Berg, DARPA Workshop, 2002] Example: IEM

  31. [Source: Berg, DARPA Workshop, 2002] Example: IEM

  32. Example: IEM [Source: Berg, DARPA Workshop, 2002] MP1-35

  33. Example: IEM [Source: Berg, DARPA Workshop, 2002] MP1-36

  34. Speed: TradeSports [Source: Wolfers 2004] Contract: Pays $100 if Cubs win game 6 (NLCS) Price of contract (=Probability that Cubs win) Fan reaches over and spoils Alou’s catch. Still 1 out. Cubs are winning 3-0 top of the 8th1 out. The Marlins proceed to hit 8 runs in the 8th inning Time (in Ireland) MP1-37

  35. Experiment 2003 NFL Season ProbabilitySports.com Online football forecasting competition Contestants assess probabilities for each game Quadratic scoring rule ~2,000 “experts”, plus: NewsFutures (play $) Tradesports (real $) Used “last trade” prices Results: Play money and real money performed similarly 6th and 8th respectively Markets beat most of the ~2,000 contestants Average of experts came 39th (caveat) Does money matter? Play vs real, head to head Electronic Markets, Emile Servan-Schreiber, Justin Wolfers, David Pennock and Brian Galebach

  36. Does money matter? Play vs real, head to head Statistically:TS ~ NFNF >> Avg TS > Avg

  37. Real marketsvs. market games IEM HSX averagelog score arbitrageclosure MP1-41

  38. Real marketsvs. market games probabilisticforecasts HSX FX, F1P6 forecast source avg log score F1P6 linear scoring -1.84 F1P6 F1-style scoring -1.82 betting odds -1.86 F1P6 flat scoring -2.03 F1P6 winner scoring -2.32 expectedvalueforecasts489 movies MP1-42

  39. 1/7 Story Survey Research A Wisdom of Crowds Story Opinion • ProbabilitySports.com • Thousands of probability judgments for sporting events • Alice: Jets 67% chance to beat Patriots • Bob: Jets 48% chance to beat Patriots • Carol, Don, Ellen, Frank, ... • Reward: Quadratic scoring rule:Best probability judgments maximize expected score

  40. Individuals • Most individuals are poor predictors • 2005 NFL Season • Best: 3747 points • Average: -944 Median: -275 • 1,298 out of 2,231 scored below zero(takes work!)

  41. Poorly calibrated (too extreme) Teams given < 20% chance actually won 30% of the time Teams given > 80% chance actually won 60% of the time Individuals

  42. The Crowd • Create a crowd predictor by simply averaging everyone’s probabilities • Crowd = 1/n(Alice + Bob + Carol + ... ) • 2005: Crowd scored 3371 points(7th out of 2231) ! • Wisdom of fools: Create a predictor by averaging everyone who scored below zero • 2717 points (62nd place) ! • (the best “fool” finished in 934th place)

  43. The Crowd: How Big? More:http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/http://www.overcomingbias.com/2007/02/how_and_when_to.html

  44. Can We Do Better?: ML/Stats [Dani et al. UAI 2006] • Maybe Not • CS “experts algorithms” • Other expert weights • Calibrated experts • Other averaging fn’s (geo mean, RMS, power means, mean of odds, ...) • Machine learning (NB, SVM, LR, DT, ...) • Maybe So • Bayesian modeling + EM • Nearest neighbor (multi-year)

  45. Can we do better?: Markets

More Related