Collective Revelation: A Mechanism for Self-Verified, Weighted, and Truthful Predictions

1. Collective Revelation: A Mechanism for Self-Verified,Weighted, and Truthful Predictions SharadGoel, Daniel M. Reeves, David M. Pennock Presented by: Nir Shabbat

2. Outline • Introduction • Background and Related Work • The Settings • A Mechanism For Collective Revelation • The Basic Mechanism • A General Technique for Balancing Budgets • Summary

3. Introduction • In many cases, a decision maker may seek to elicit and aggregate the opinions of multiple experts. • Ideally, would like a mechanism that is: • Incentive Compatible - Rewards participants to be truthful • Information Weighted - Adjusts for the fact that some experts are better informed than others • Self-Verifying - Works without the need for objective, “ground truth” observations • Budget Balanced - Makes no net transfers to agent

4. Background and Related Work • Proper Scoring Rules – Rewards the agent by assessing his forecast against an actual observed outcome. • Agents are incentivized to be truthful. • For example, using the Brier scoring rule: We get:

5. Background and Related Work • Peer PredictionMethods – Can be Incentive Compatible and Self-Verifying. • For example, the Bayesian truth serum asks agents to report both their own prediction and their prediction of other agents’ predictions. • Relies on the general property of information- scores, that a truthful answer constitutes the best guess about the most “surprisingly common” answer.

6. Background and Related Work • With the appropriate reward structure , this framework leads to truthful equilibria: • iff response of agent is • - prediction of player for the frequency of • - geometric average of predictions (or opinions) and meta predictions

7. Background and Related Work • Predictions Markets – encourage truthful behavior and automatically aggregate predictions from agents with diverse information. • For example, a prediction market for U.S. presidential elections. Agents buy and sell assets tied to an eventual Democratic or Republican win. Each share pay $1 if the corresponding event occurs.

8. Background and Related Work • Delphi Method – generates consensus predictions from experts generally through a process of structured discussion. • In each round, each expert anonymously provides his forecast and reasons, at the end of the round, market maker summarize the forecasts and reasons. • Process is repeated until forecasts converge.

9. Background and Related Work • Competitive Forecasting – elicits confidence intervals on predictions, thereby facilitating information weighting. Like prediction markets, competitive forecasting rewards accuracy, though is not rigorously incentive compatible and relies on benchmarking against objective measurements.

10. Background and Related Work

11. The Settings • Common prior – Agents & Market Maker have a common prior on the distribution of • Independent Private Evidence – Each agent privately observes independent realization of the random variable (independent both of each other and across agents). • Rationality– Agents update their beliefs via Bayes’ rule. • Risk neutrality– Agents act to maximize their expected payoff.

12. A Mechanism For Collective Revelation • Agents are scored against one another’s private information (Self Verification). • Agents report their subjective expectation of , and their revised estimate in light of new hypothetical evidence (HFS). • From these 2 reports the subjective posterior, private information and also this agent’s prediction of other agents’ prediction is constructed.

13. A Mechanism For Collective Revelation

14. The Basic Mechanism LEMMA 1 • An agent has subjective distribution for It is possible to elicit the kth moment of via a proper scoring rule that pays based on the outcome iff

15. The Basic Mechanism • Simply put, scoring against a single Bernoulli observation can only reveal the agent’s expectation and not the uncertainty of it’s prediction (as quantified by the variance). • So in order for a mechanism to be Information-Weighted (for Bernoulli outcome) it must score agents against multiple observations.

16. The Basic Mechanism • For example, say we want to elicit an agent’s prediction regarding the distribution of . • We want to reward the agent using a proper scoring rule based on a single outcome : • To elicit the confidence, we must use another outcome . For example, like this:

17. The Basic Mechanism The Beta distribution is the “conjugate prior” of the Bernoulli distribution. That is, given with a prior the posterior distribution of , given independent trials out of which are successful, is:

18. The Basic Mechanism LEMMA 2 • Let • a prior on • Fix (trials) and (successes) integers • Define: That is:

19. The Basic Mechanism LEMMA 2 (Cont’d) Then: • is a bijection from to • And

20. The Basic Mechanism LEMMA 3 • Let • Let a prior on • Let • Define:

21. The Basic Mechanism LEMMA 3 (Cont’d) Then: • is a bijection from to • And:

22. The Basic Mechanism THEOREM 1 • Consider a Bayesian game according to the settings shown before with players, . • Let a common prior on • Let be the posterior dist. of agent for after updating according to its private info. • Fix and integers

23. The Basic Mechanism THEOREM 1 (Cont’d) • Suppose that each agent plays action where

24. The Basic Mechanism THEOREM 1 (Cont’d) • Define: Where is projection into the kth component.

25. The Basic Mechanism THEOREM 1 (Cont’d) • For arbitrary constants , set the reward of agent to be:

26. The Basic Mechanism THEOREM 1 (Cont’d) • Then: (*) is a strict Nash Equilibrium.

27. The Basic Mechanism PROOF • Fix attention on agent , and suppose all agents play according to (*). • By Lemma 2 - parameters of .

28. The Basic Mechanism PROOF (Cont’d) • Let be private observations of agent . Then: Consequently:

29. The Basic Mechanism PROOF (Cont’d) • In particular, and • If plays according to (*) then: Since we are using the Brier Proper scoring rule, strategy (*) maximizes ’s expected reward.

30. The Basic Mechanism PROOF (Cont’d) • Moreover, since is an injection, this is ’s unique best response, and so strategy (*) is a strict Nash equilibrium.

31. The Basic Mechanism CORROLARY • Suppose all agents play the equilibrium strategy. • Define:

32. The Basic Mechanism CORROLARY (Cont’d) • Define the aggregate information-weighted prediction as: Let be the posterior dist. Resulting from the cumulative private evidence of all agents. Then .

33. A General Technique for Balancing Budgets • The idea is to reward agents via a Shared scoring rule. A simple application would be: And this is clearly budget balanced since

34. A General Technique for Balancing Budgets • In our setting, the “observations” are determined precisely by agents’ reports – simple application won’t work! • The solution proposed is to decouple scoring from benchmarking, e.g. agent is rewarded according to its performance relative to a set of agents whose score cannot be affected by agent ’s action.

35. A General Technique for Balancing Budgets DEFINITION • Let be a Bayesian game with players • A -projective family of is a family of games such that for each with , is a Bayesian game restricted to the players that preserves type spaces, action spaces, and players’ beliefs about types. In particular, is determined by a family of reward functions that specifies the expected reward for each player in resulting from any given strategy profile of those players.

36. A General Technique for Balancing Budgets THEOREM 2 • G is a Bayesian game with n players • is a (strict) Nash equilibrium of G • Suppose is a k-projective family of Gsuch that • Suppose that for each , is a (strict) Nash equilibrium for .

37. A General Technique for Balancing Budgets THEOREM 2 (Cont’d) • Then for any constant , there are player rewards for the n-player game G such that: • For any strategy profile s, • The original equilibrium q is still a (strict) Nash equilibrium for the modified game In particular, by setting , one can alter the rewards so that the game G is strongly budget balanced.

38. A General Technique for Balancing Budgets PROOF • For , denote: • - reward function of player in • - reward function of player in the game • - reward of player in game , when is restricted to game .

39. A General Technique for Balancing Budgets PROOF (Cont’d) • Denote the player sets where the players wrap around for . • Consider the modified rewards for :

40. A General Technique for Balancing Budgets PROOF (Cont’d) • Summing over all players we get:

41. A General Technique for Balancing Budgets PROOF (Cont’d) • Further more: And since , we have , and so for .

42. A General Technique for Balancing Budgets PROOF (Cont’d) • Consequently, if is the strategy of player and is the strategy profile of all other players, then: Where is a function that does not depend on .

43. A General Technique for Balancing Budgets PROOF (Cont’d) • Since is a (strict) Nash equilibrium for : And so, using the new reward functions, the original equilibrium is still a (strict) Nash equilibrium.

44. Summary • We’ve seen a mechanism, i.e. Collective Revelation, for aggregating experts opinions which is: • Incentive Compatible • Information Weighed • Self-Verifying • We’ve seen a general technique for constructing budget balanced mechanisms that applies both to collective revelation and to past peer-predictions method.

45. THE END