1 / 54

Prediction Markets and the Wisdom of Crowds

Prediction Markets and the Wisdom of Crowds. David Pennock , Yahoo! Research Joint with:

samara
Download Presentation

Prediction Markets and the Wisdom of Crowds

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Prediction Markets and the Wisdom of Crowds David Pennock, Yahoo! Research Joint with: Yiling Chen, Varsha Dani, Lance Fortnow, Ryan Fugger, Brian Galebach, Arpita Ghosh, Sharad Goel, Mingyu Guo, Joe Kilian, Nicolas Lambert, Omid Madani, Mohammad Mahdian, Eddie Nikolova, Daniel Reeves, Sumit Sanghai, Mike Wellman, Jenn Wortman

  2. Bet = Credible Opinion Obama will win the 2008 US Presidential election • Which is more believable?More Informative? • Betting intermediaries • Las Vegas, Wall Street, Betfair, Intrade,... • Prices: stable consensus of a large number of quantitative, credible opinions • Excellent empirical track record “I bet $100 Obama will win at 1 to 2 odds”

  3. A Prediction Market • Take a random variable, e.g. • Turn it into a financial instrument payoff = realized value of variable Bin Laden captured in 2008?(Y/N) I am entitled to: Bin Ladencaught ’08 Bin Ladencaught ’08 $1 if $0 if

  4. http://intrade.com

  5. Outline • The Wisdom of Crowds • The Wisdom of Markets • Prediction Markets:Examples & Research • Does Money Matter? • Combinatorial Betting Story Survey Research Research

  6. 1/7 Story Survey Research A WOC Story Opinion • ProbabilitySports.com • Thousands of probability judgments for sporting events • Alice: Jets 67% chance to beat Patriots • Bob: Jets 48% chance to beat Patriots • Carol, Don, Ellen, Frank, ... • Reward: Quadratic scoring rule:Best probability judgments maximize expected score

  7. Individuals • Most individuals are poor predictors • 2005 NFL Season • Best: 3747 points • Average: -944 Median: -275 • 1,298 out of 2,231 scored below zero(takes work!)

  8. Poorly calibrated (too extreme) Teams given < 20% chance actually won 30% of the time Teams given > 80% chance actually won 60% of the time Individuals

  9. The Crowd • Create a crowd predictor by simply averaging everyone’s probabilities • Crowd = 1/n(Alice + Bob + Carol + ... ) • 2005: Crowd scored 3371 points(7th out of 2231) ! • Wisdom of fools: Create a predictor by averaging everyone who scored below zero • 2717 points (62nd place) ! • (the best “fool” finished in 934th place)

  10. The Crowd: How Big? More:http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/http://www.overcomingbias.com/2007/02/how_and_when_to.html

  11. Can We Do Better?: ML/Stats [Dani et al. UAI 2006] • Maybe Not • CS “experts algorithms” • Other expert weights • Calibrated experts • Other averaging fn’s (geo mean, RMS, power means, mean of odds, ...) • Machine learning (NB, SVM, LR, DT, ...) • Maybe So • Bayesian modeling + EM • Nearest neighbor (multi-year)

  12. Can we do better?: Markets

  13. Prediction Markets:Examples & Research

  14. What you can say/learn% chance that Obama wins GOP wins Texas YHOO stock > 30 Duke wins tourney Oil prices fall Heat index rises Hurricane hits Florida Rains at place/time Where IEM, Intrade.com Intrade.com Stock options market Las Vegas, Betfair Futures market Weather derivatives Insurance company Weatherbill.com The Wisdom of CrowdsBacked in dollars

  15. Prediction MarketsWith Money Without

  16. The Widsom of CrowdsBacked in “Points” • HSX.com • Newsfutures.com • InklingMarkets.com • Foresight Exchange • CasualObserver.net • FTPredict.com • Yahoo!/O’Reilly Tech Buzz • ProTrade.com • StorageMarkets.com • TheSimExchange.com • TheWSX.com • Alexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ, MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds • http://www.chrisfmasse.com/3/3/markets/#Play-Money_Prediction_Markets

  17. http://betfair.com Screen capture 2008/05/07 http://tradesports.com Screen capture 2007/05/18

  18. [Source: Berg, DARPA Workshop, 2002] Example: IEM 1992

  19. [Source: Berg, DARPA Workshop, 2002] Example: IEM

  20. [Source: Berg, DARPA Workshop, 2002] Example: IEM

  21. [Thanks: Yiling Chen] Does it work? • Yes, evidence from real markets, laboratory experiments, and theory • Racetrack odds beat track experts [Figlewski 1979] • Orange Juice futures improve weather forecast [Roll 1984] • I.E.M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002] • HP market beat sales forecast 6/8 [Plott 2000] • Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’03][Schmidt 2002] • Laboratory experiments confirm information aggregation[Plott 1982;1988;1997][Forsythe 1990][Chen, EC’01] • Theory: “rational expectations” [Grossman 1981][Lucas 1972] • Market games work [Servan-Schreiber 2004][Pennock 2001]

  22. Prediction Markets:Does Money Matter?

  23. IEM: 237 Candidates HSX: 489 Movies The Wisdom of CrowdsWith Money Without

  24. The Wisdom of CrowdsWith Money Without

  25. probabilisticforecasts Real markets vs. market games HSX FX, F1P6 forecast source avg log score F1P6 linear scoring -1.84 F1P6 F1-style scoring -1.82 betting odds -1.86 F1P6 flat scoring -2.03 F1P6 winner scoring -2.32

  26. Experiment 2003 NFL Season ProbabilitySports.com Online football forecasting competition Contestants assess probabilities for each game Quadratic scoring rule ~2,000 “experts”, plus: NewsFutures (play $) Tradesports (real $) Used “last trade” prices Results: Play money and real money performed similarly 6th and 8th respectively Markets beat most of the ~2,000 contestants Average of experts came 39th (caveat) Does money matter? Play vs real, head to head Electronic Markets, Emile Servan-Schreiber, Justin Wolfers, David Pennock and Brian Galebach

  27. Does money matter? Play vs real, head to head Statistically:TS ~ NFNF >> Avg TS > Avg

  28. Discussion • Are incentives for virtual currency strong enough? • Yes (to a degree) • Conjecture: Enough to get what people already know; not enough to motivate independent research • Reduced incentive for information discovery possibly balanced by better interpersonal weighting • Statistical validations show HSX, FX, NF are reliable sources for forecasts • HSX predictions >= expert predictions • Combining sources can help

  29. A Problem w/ Virtual CurrencyPrinting Money Alice1000 Betty1000 Carol1000

  30. A Problem w/ Virtual CurrencyPrinting Money Alice5000 Betty1000 Carol1000

  31. YootlesA Social Currency Alice0 Betty0 Carol0

  32. YootlesA Social Currency I owe you 5 Alice-5 Betty0 Carol5

  33. YootlesA Social Currency I owe you 5 credit: 5 credit: 10 Alice-5 Betty0 Carol5

  34. YootlesA Social Currency I owe you 5 I owe you 5 credit: 5 credit: 10 Alice-5 Betty0 Carol5

  35. YootlesA Social Currency I owe you 5 I owe you 5 credit: 5 credit: 10 Alice3995 Betty0 Carol5

  36. YootlesA Social Currency • For tracking gratitude among friends • A yootle says “thanks, I owe you one”

  37. Combinatorial Betting

  38. Combinatorics ExampleMarch Madness

  39. Typical todayNon-combinatorial Team wins Rnd 1 Team wins Tourney A few other “props” Everything explicit(By def, small #) Every bet indep: Ignores logical & probabilistic relationships Combinatorial Any property Team wins Rnd kDuke > {UNC,NCST}ACC wins 5 games 2263 possible props(implicitly defined) 1 Bet effects related bets “correctly”;e.g., to enforce logical constraints Combinatorics ExampleMarch Madness

  40. Expressiveness:Getting Information • Things you can say today: • (63% chance that) Obama wins • GOP wins Texas • YHOO stock > 30 Dec 2007 • Duke wins NCAA tourney • Things you can’t say (very well) today: • Oil down, DOW up, & Obama wins • Obama wins election, if he wins OH & FL • YHOO btw 25.8 & 32.5 Dec 2007 • #1 seeds in NCAA tourney win more than #2 seeds

  41. Expressiveness:Processing Information • Independent markets today: • Horse race win, place, & show pools • Stock options at different strike prices • Every game/proposition in NCAA tourney • Almost everything: Stocks, wagers, intrade, ... • Information flow (inference) left up to traders • Better: Let traders focus on predicting whatever they want, however they want: Mechanism takes care of logical/probabilistic inference • Another advantage: Smarter budgeting

  42. A > B > C .1 A > C > B .2 B > A > C .1 B > C > A .3 C > A > B .1 C > B > A .2 Market CombinatoricsPermutations

  43. D > A > B > C .01 D > A > C > B .02 D > B > A > C .01 A > D > B > C .01 A > D > C > B .02 B > D > A > C .05 A > B > D > C .01 A > C > D > B .2 B > A > D > C .01 A > B > C > D .01 A > C > B > D .02 B > A > C > D .01 D > B > C > A .05 D > C > A > B .1 D > C > B > A .2 B > D > C > A .03 C > D > A > B .1 C > D > B > A .02 B > C > D > A .03 C > A > D > B .01 C > B > D > A .02 B > C > D > A .03 C > A > D > B .01 C > B > D > A .02 Market CombinatoricsPermutations

  44. Bidding Languages • Traders want to bet on properties of orderings, not explicitly on orderings: more natural, more feasible • A will win ; A will “show” • A will finish in [4-7] ; {A,C,E} will finish in top 10 • A will beat B ; {A,D} will both beat {B,C} • Buy 6 units of “$1 if A>B” at price $0.4 • Supported to a limited extent at racetrack today, but each in different betting pools • Want centralized auctioneer to improve liquidity & information aggregation

  45. Example • A three-way match • Buy 1 of “$1 if A>B” for 0.7 • Buy 1 of “$1 if B>C” for 0.7 • Buy 1 of “$1 if C>A” for 0.7 B A C

  46. Pair Betting • All bets are of the form “A will beat B” • Cycle with sum of prices > k-1 ==> Match(Find best cycle: Polytime) • Match =/=> Cycle with sum of prices > k-1 • Theorem: The Matching Problem for Pair Betting is NP-hard (reduce from min feedback arc set)

  47. [Thanks: Yiling Chen] Automated Market Makers • A market maker (a.k.a. bookmaker) is a firm or person who is almost always willing to accept both buy and sell orders at some prices • Why an institutional market maker? Liquidity! • Without market makers, the more expressive the betting mechanism is the less liquid the market is (few exact matches) • Illiquidity discourages trading: Chicken and egg • Subsidizes information gathering and aggregation: Circumvents no-trade theorems • Market makers, unlike auctioneers, bear risk. Thus, we desire mechanisms that can bound the loss of market makers • Market scoring rules [Hanson 2002, 2003, 2006] • Dynamic pari-mutuel market [Pennock 2004]

  48. Overview: Complexity Results

  49. March Madness bet constructor Bet on any team to win any game Duke wins in Final 4 Bet “exotics”: Duke advances further than UNC ACC teams win at least 5 A 1-seed will lose in 1st round

More Related