An Experimental Test of Combinatorial Information Markets

1. An Experimental Test ofCombinatorial Information Markets Robin Hanson George Mason University

2. “Info” or “Bit” Economy? • Most “info” in info economy is just bits • Music, movies, gossip, news, chat, … • The info we really want: • Productive? Generous? Loyal? Good genes? How you do that so well? How cure me? • In this sense, always been info economy • Easy to see what people say, but how to see what they really think?

3. Information Markets • Most speculative markets aggregate info well • Very hard to find info to beat average return • In direct comparisons, beat other institutions • Racetrack odds beat track experts (Figlewski 1979) • OJ futures improve weather forecast (Roll 1984) • Gas demand markets beat experts (Spencer 2004) • I.E.M. beat president polls 451/596 (Berg et al 2001) • HP market beat sales forecast 6/8 (Plott 2000) • Stocks beat Challenger panel (Maloney & Mulherin 2003)

4. “Pays $1 if Bush wins” Will price rise or fall? sell E[ price change | ?? ] buy price sell Lots of ?? get tried, price includes all! buy Buy Low, Sell High (All speculation is “gambling”!)

5. Today’s Prices 66-67%  Aviator Oscar best picture 2004 15% Bin Laden captured by 6/30/05 71-73% Michael Jackson guilty lewd acts 10-12% Palestinian State before 2006 36-38% Hillary Demo. candidate in 2008 18% McCain Rep. candidate in 2008 50-51%  Democrat President in 2008 TradeSports.com

6. Info Markets Benefits • People self-select as experts • Incentive to be honest with yourself, and to shut up if you don’t know • If not honest, eventually lose $, then shut up. • Precise and continually updated • Consistent across diverse contexts • Can specialize, correct biases see

7. Troop Move Decision Advice $1 if War & Move Troops P(M) $1 if Move Troops P(W | M) $1 Compare! P(W | not M) $1 if Not Move Troops $1 if War & Not Move Troops P(not M)

8. Decision Market Applications E[ firm stock price | fire CEO? ] E[ f(inflate,unemploy) | Fed raise rates? ] E[ USA lifespan | national health insurance ] E[ my years to live | opt for surgery? ] E[ crime rate | gun control bill pass? ] E[ Democrat win | Nominate Hillary? ] E[ GDP | Demo. president 2008? ] E[ Iraq civil war | US moves troops out? ]

10. We Wanted: • Every nation*quarter: • Political stability • Military activity • Economic growth • US $ aid • US military activity • & global, special • & all combinations

11. Return to Focus ? Trade IQcs4 IQcs4 < 85 85 03 03 SAum3 105-125 03 Update Payoffs: If & Ave. pay Select New Price 65% Max Up 95.13% +$34.74 -$85.18 -$19.72 Buy 10% Up 68.72% +$2.74 -$3.28 -$1.07 You Pick 65 % +1.43 -2.04 +0.34 Saudi Arabian Economic Health No Trade 62.47% $0.00 $0.00 $0.00 125 30 15 10% Dn 56.79% -$2.61 +$2.74 -$1.12 65 70 Sell Exit Issue 48.54% -$15.34 +$26.02 -$6.31 35 40 100 94 100 Max Dn 22.98% -$120.74 +$96.61 -$22.22 < 85 25 35 35 30 10 10 75 1 2 3 4 1 2 > 03 03 03 03 04 04 ? Return to Form Execute a Trade If US military involvement in Saudi Arabia in 3rd Quarter 2003 is not between 105 and 125, this trade is null and void. Otherwise, if Iraq civil stability in 4th Quarter 2003 is below 85, then I will receive $1.43, but if it is not below 85, I will pay $2.04. Abort trade if price has changed? Execute Trade Scenario

12. Thin Market Problem • Trade requires coordinate in Assets and • Time: waiting offers suffer adverse selection • Call markets, combo match, can help some, but • Most possible info markets do not exist • Most illegal, if expect few traders don’t make offer • If known that only one person has opinion on a topic, price of simple market not reveal it • Key cost driver: need #traders > #assets

13. Simple Info Markets Market Scoring Rules Scoring Rules opinion pool problem thin market problem 100 .001 .01 .1 1 10 Best of Both Accuracy Estimates per trader

14. Market Scoring Rules • MSRs combine scoring rules, info markets • “Anyone can use scoring rule if pay off last user” • Is auto market maker, takes any combo bet • No spread if tiny trade, price worse with quantity • Needs subsidy per base var, more for thicker • No extra subsidy for all time, all combos! • Logarithmic version is nicely modular • Bet on p(A|B) changes it, but not p(B), p(C|A&B), p(C|A&B), p(C|B),I(A,B,C), I(B,A,C)

15. $1 if A&B p(A|B) $1 if B Log MSR Cost & Modularity • Cost = strue(pend) - strue(pstart) • Expected cost: Ep[C] Sipi (si(1i) - si(p)) • Log cost bound  entropy: S(p) = - Sipi log(pi) • S(pall)Svar S(pvar), so all combos  rule per var! • Log is modular • Changes p(A|B), but not p(B), p(C|A&B), p(C|A&B), p(C|B),I(A,B,C), I(B,A,C)

16. Laboratory Tests • Joint work with John Ledyard (Caltech), Takashi Ishida (Net Exchange) • Caltech students, get ~$30/session • 6 periods/session, 12-15 minutes each • Trained in 3var session, return for 8var • Metric: Kulback-Leibleri qilog(pi /qi) distance from mechanism estimate to Bayesian beliefs given all group info

17. Environments: Goals, Training (Actually: X Z Y ) Case A B C 1 1 - 1 2 1 - 0 3 1 - 0 4 1 - 0 5 1 - 0 6 1 - 1 7 1 - 1 8 1 - 0 9 1 - 0 10 0 - 0 Sum: 9 - 3 Same A B C A -- -- 4 B -- -- -- C -- -- -- • Want in Environment: • Many vars, few related, guess which • Use both theory and data • Few people, specialize in variable sets • Can compute rational group estimates • Explainable, fast, neutral • Training Environment: • 3 binary variables X,Y,Z, 23 = 8 combos • P(X=0) = .3, P(X=Y) = .2, P(Z=1)= .5 • 3 people, see 10 cases of: AB, BC, AC • Random map XYZ to ABC

18. Theory Benchmarks 3 Variables

19. Simple Double Auction 3 Variables

20. Combinatorial Call 3 Variables

21. Individual Scoring Rule 3 Variables

22. Best of Three Individuals 3 Variables

23. Log Opinion Pool 3 Variables

24. Market Scoring Rule 3 Variables

25. KL(prices,group) 1- KL(uniform,group) MSR Info vs. Time – 3 Variables 1 % Info Agg. = 0 0 5 10 15 Minutes -1

26. Experiment Environment (Really: W V X S U Z Y T ) Case A B C D E F G H 1 0 1 0 1 - - - - 2 1 0 0 1 - - - - 3 0 0 1 1 - - - - 4 1 0 1 1 - - - - 5 0 1 1 1 - - - - 6 1 0 0 1 - - - - 7 0 1 1 1 - - - - 8 1 0 0 1 - - - - 9 1 0 0 1 - - - - 10 1 0 0 1 - - - - Sum 6 3 4 10 - - - - Same A B C D E F G H A -- 1 2 6 -- -- -- -- B -- -- 7 3 -- -- -- -- C -- -- -- 4 -- -- -- -- D -- -- -- -- -- -- -- -- … • 8 binary vars: STUVWXYZ • 28 = 256 combinations • 20% = P(S=0) = P(S=T) = P(T=U) = P(U=V) = … = P(X=Y) = P(Y=Z) • 6 people, each see 10 cases: ABCD, EFGH, ABEF, CDGH, ACEG, BDFH • random map STUVWXYZ to ABCDEFGH

27. Theory Benchmarks 8 Variables

28. Simple Double Auction 8 Variables

29. Combinatorial Call 8 Variables

30. Individual Scoring Rule 8 Variables

31. Market Scoring Rule 8 Variables

32. Log Opinion Pool 8 Variables

33. Best of Three Individuals 8 Variables

34. KL(prices,group) 1- KL(uniform,group) MSR Info vs. Time – 8 Variables 1 % Info Agg. = 0 0 5 10 15 Minutes -1

35. Market Maker Histogram – 3 Vars Log(KL distance)

36. Accuracy 95% Confidence Levels

37. Experiment Conclusions • Experiments on complex info problems • Bayesian estimates far too high a standard • 7 indep. prices from 3 people in 5 minutes • Simple DA < Indiv. Score Rule ~ Opinion Pool ~ Combined Value < Market Scoring Rule • 255 indep. prices from 6 people in 5 minutes • Combined Value ~ Simple DA ~ Indiv. Score Rule < Market Scoring Rule ~ Opinion Pool

38. MSR Scaling Issues D A C G F B • Simple: store 2N states • Feasible: overlapping var patches • Simple MSR per patch • Allow trade only if all vars in same patch • Users share assets across patches • How make patches agree? • Arbitrage neighbors is robust, non-modular • Bayes Net modular, but is money pump? E H

39. Sweepstake Sports Bet Racetrack Lottery Casino Regulation of Speculation Gamble Capitalize Firms Hedge Individual Risk “Entertain” Hedge Common Risk Aggregate Information

40. Timeline - Policy Analysis Market 2001 May – DARPA seeks info market proposals Fall – We start Phase I ($100K+$100K) Poindexter heads IAO, we run experiments Spring – We start Phase II ($750K) May – Congress report ex: Israel Bio Terror Plan: register 8/1, few test 9/1, open 1/1 7/28 – Senators complain: “repugnant" 7/29 – DARPA FutureMAP Killed 7/31 – Poindexter quits 2002 2003

41. The Fuss: Analysts often use prices from various markets as indicators of potential events. The use of petroleum futures contract prices by analysts of the Middle East is a classic example. The Policy Analysis Market (PAM) refines this approach by trading futures contracts that deal with underlying fundamentals of relevance to the Middle East. Initially, PAM will focus on the economic, civil, and military futures of Egypt, Jordan, Iran, Iraq, Israel, Saudi Arabia, Syria, and Turkey and the impact of U.S. involvement with each. [Click here for a summary of PAM futures contracts] The contracts traded on PAM will be based on objective data and observable events. These contracts will be valuable because traders who are registered with PAM will use their money to acquire contracts. A PAM trader who believes that the price of a specific futures contract under-predicts the future status of the issue on which it is based can attempt to profit from his belief by buying the contract. The converse holds for a trader who believes the price is an over-prediction – she can be a seller of the contract. This price discovery process, with the prospect of profit and at pain of loss, is at the core of a market’s predictive power. The issues represented by PAM contracts may be interrelated; for example, the economic health of a country may affect civil stability in the country and the disposition of one country’s military may affect the disposition of another country’s military. The trading process at the heart of PAM allows traders to structure combinations of futures contracts. Such combinations represent predictions about interrelated issues that the trader has knowledge of and thus may be able to make money on through PAM. Trading these trader-structured derivatives results in a substantial refinement in predictive power. [Click here for an example of PAM futures and derivatives contracts] The PAM trading interface presents A Market in the Future of the Middle East. Trading on PAM is placed in the context of the region using a trading language designed for the fields of policy, security, and risk analysis. PAM will be active and accessible 24/7 and should prove as engaging as it is informative.

42. Is Terror Betting “Unthinkable”? • We deny trading sacred/secular (P. Tetlock) • E.g., money vs. death risk • Outraged at those we see do (really, we all do) • More outraged if they think a long time first • If catch ourselves, seek moral cleansing • PAM painted as crossing a moral boundary • “None of us should intend to benefit when they hurt some of us.” • So politicians must not think long before rejecting

43. Possible Manipulation • Bad guys gain $ by giving info, changing acts • More plausible if bet on specific details • A good deal for us if give few $, gain much info • Terror now effects big markets: oil, currency, … • PAM not on specifics, max gain/trade < $100 • Bad guys lose $ to obscure market info • Worst case: get no info (they can’t make net bias) • $1M worth it if 0.1% chance gain 0.1% of $400B/yr • Usually “noise traders” attract others, net add info • We see little effect in lab, field experiments

44. Why Not Only Experts, Or Play $? • Must restrict to experts if need prices secret • No direct tests yet on open/closed, play/real $ • Main success record is for open trading, real $ • Open markets over-weigh experts (Figlewski 1979, Metzger 1985, Lichtenstein, Kaufmann & Bhagat 1999) • Their best traders have average IQ (Ceci & Liker, 1986) • PAM needed lots of trading data in 2 years • Very hard to move $ between govt. agencies • No single agency very enthusiastic • Funding limits prevented run in parallel to compare

45. Price Manipulation Model Kyle Style Market Microstructure Market maker Manipulator Informed trader Noise trader Equilibrium

46. Decision Selection Bias • If traders think deciders will use info traders do not have, conditional market price advice may contradict trader info • Related to “Newcomb’s Problem” in decision theory

47. Best to keep in this case Stock if keep CEO Better to dump Best to dump in this case Expected value over distribution is center of mass Stock if dump CEO A Decision State Space Imagine a uniform distribution over this area

48. If No Selection Bias Market prices here if decision not correlated with state Stock if keep CEO Better to dump Stock if dump CEO

49. Well-Informed Deciders Keep Stock if keep CEO Apparent center Dump True center Stock if dump CEO

50. Problem Seems Uncommon