1 / 49

Information Aggregation: Experiments and Industrial Applications

Information Aggregation: Experiments and Industrial Applications. Kay-Yut Chen HP Labs. Agenda. Lessons from HP Information Markets (Chen and Plott 2002) Scoring Rules and Identification of Experts (Chen, Fine and Huberman 2004) (Chen and Hogg 2004) Public Information

darryldavis
Download Presentation

Information Aggregation: Experiments and Industrial Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Information Aggregation:Experiments and Industrial Applications Kay-Yut Chen HP Labs

  2. Agenda • Lessons from HP Information Markets (Chen and Plott 2002) • Scoring Rules and Identification of Experts (Chen, Fine and Huberman 2004) (Chen and Hogg 2004) • Public Information (Chen, Fine and Huberman 2004) Experimental Economics Program

  3. HP Information Markets (Chen and Plott) • Summary of Events • 12 events, from 1996 to 1999 • 11 events sales related • 8 events had official forecasts • Methodology & Procedures • Contingent state asset (i.e. winning ticket pays $1, others $0) • Sales amount (unit/revenue) divided into (8-10) finite intervals • Web-based real time double-auction • 15-20 min phone training for EVERY subject • Market open for one week at restricted time of the day (typically lunch and after hours) • Market size: 10-25 people Experimental Economics Program

  4. Experimental Economics Program

  5. Results Experimental Economics Program

  6. Business Constraints and Research Issues • Not allowed to “bet” players’ own money -> stakes limited to an average of $50 per person • Time horizon constraints -> 3 months to be useful • Recruit the “right” people • Asset design affects the results (How to set the intervals?) • Thin markets (sum of price ~ $1.11 to $1.31 over the dollar) • Few players • Not enough participation Experimental Economics Program

  7. Reporting with Scoring Rule Outcome A B C Reports of Probability Distribution p1 p2 p3 Pays C1+C2*Log(p3) Experimental Economics Program

  8. Information Aggregation Function If reports are independent, Bayes Law applies … Experimental Economics Program

  9. Two Complications • Non-Risk Neutral Behavior • Public Information Experimental Economics Program

  10. Dealing with Risks Attitudes:Two-Stage Mechanism Stage 1: Information Market Call Market to Solicit Risk Attitudes Time Event 1 Event 2 Event 3 Event 4 Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Nonlinear Aggregated Function Event 5 Event 6 Event 7 Event 8 Experimental Economics Program

  11. Second Stage: Aggregation Function Bayes Law with Behavioral Correction Normalizing constant for individual risks i=r(V i / i)c Holding value/Risk - measure relative risk of individuals “market” risk ~sum of prices/winning payoff Experimental Economics Program

  12. Box of Balls A C C B C Experiments:Inducing Diverse Information Outcome A B C Random Draws Provide Info * In actual experiments, there are TEN states Experimental Economics Program

  13. Comparison To All Information Probability Kullback-Leibler = 1.453 Experiment 4, Period 17 No Information

  14. Kullback-Leibler Measure • Relative entropy • Always >=0 • =0 if two distributions are identical • Addictive for independent events Experimental Economics Program

  15. Comparison To All Information Probability Kullback-Leibler = 1.337 Experiment 4, Period 17 1 Player

  16. Comparison To All Information Probability Kullback-Leibler = 1.448 Experiment 4, Period 17 2 Players Aggregated

  17. Comparison To All Information Probability Kullback-Leibler = 1.606 Experiment 4, Period 17 3 Players Aggregated

  18. Comparison To All Information Probability Kullback-Leibler = 1.362 Experiment 4, Period 17 4 Players Aggregated

  19. Comparison To All Information Probability Kullback-Leibler = 0.905 Experiment 4, Period 17 5 Players Aggregated

  20. Comparison To All Information Probability Kullback-Leibler = 1.042 Experiment 4, Period 17 6 Players Aggregated

  21. Comparison To All Information Probability Kullback-Leibler = 0.550 Experiment 4, Period 17 7 Players Aggregated

  22. Comparison To All Information Probability Kullback-Leibler = 0.120 Experiment 4, Period 17 8 Players Aggregated

  23. Comparison To All Information Probability Kullback-Leibler = 0.133 Experiment 4, Period 17 9 Players Aggregated

  24. Comparison To All Information Probability Experiment 4, Period 17

  25. No Information Market Prediction Best Player Nonlinear Aggregation Function 1.977 (0.312) 1.222 (0.650) 0.844 (0.599) 0.553 (1.057) 1.501 (0.618) 1.112 (0.594) 1.128 (0.389) 0.214 (0.195) 1.689 (0.576) 1.053 (1.083) 0.876 (0.646) 0.414 (0.404) 1.635 (0.570) 1.136 (0.193) 1.074 (0.462) 0.413 (0.260) 1.640 (0.598) 1.371 (0.661) 1.164 (0.944) 0.395 (0.407) KL Measures for Private Info Experiments Experimental Economics Program

  26. Group Size Performance Experimental Economics Program

  27. Did the Markets Pick out Experts? • KL measure of all query data • Pick groups of 3 Experimental Economics Program

  28. Did Previous Queries Pick out Experts? • KL measure of second half of query data • Pick groups of 3 Experimental Economics Program

  29. Public Information • Information observed by more than one • Double counting problem Experimental Economics Program

  30. Information Aggregation with Public Information Kullback-Leibler = 2.591 Public Info Experiment 3, Period 9 11 Players Aggregated

  31. Dealing with Public Information:Add a Game to the Second Stage Stage 1: Information Market Call Market to Solicit Risk Attitudes Time Event 1 Event 2 Event 3 Event 4 Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Matching Game to Recover Public Information Modified Nonlinear Aggregated Function Event 5 Event 6 Event 7 Event 8 Experimental Economics Program

  32. Assumptions • Individuals know their public information • Private & Public Info Independent • Structure of Public Info Arbitrary Experimental Economics Program

  33. Choose player (3) by Max (match function) Player 1’s Payoff: (match function)*(C1+C2*Log(q33)) Match function: f(q1,q2)=(1-0.5*sum(abs(q1i-q2i)))^2 Matching Game Outcome A B C Reports of Probability Distribution q11 q12 q13 Player 1: q1 q21 q22 q23 Player 2: q2 q31 q32 q33 Player 3: q3 . . . . . . . . . . . . Experimental Economics Program

  34. Matching Game • Any match function f(q1,q2) with property • Max when q1=q2 • Multiple Equilibria • Payoff increases as entropy decreases • Hopefully, individuals report public information Experimental Economics Program

  35. Aggregation Function withPublic Information Correction Bayes Law with a) Behavioral Correction b) Public Info Correction Normalizing constant for individual risks i=r(Vi /i)c Holding value/Risk - measure relative risk of individuals “market” risk ~sum of prices/winning payoff Experimental Economics Program

  36. Public Information Experiments • 5 Experiments • Various Information Structures • All subject received 2 private draws & 2 public draws • All subject received 3 private draws & 1 public draws • All subject received 3 private draws & half of the subjects receive 1 public draws • All subject received 3 private draws & 1 public draws. 2 groups of independent public information. • 9 to 11 participants in each experiments Experimental Economics Program

  37. Correcting for Public Information Kullback-Leibler = 0.291 Public Info Experiment 3, Period 9 11 Players Aggregated

  38. Expt Private Info Public Info No Info Market Prediction Best Player Nonlinear Aggregation Function Public Info Correction Perfect Public Info Correction 1 2 draws for all 2 draws for all 1.332 (0.595) 0.847 (0.312) 0.932 (0.566) 2.095 (1.196) 0.825 (0.549) 0.279 (0.254) 2 2 draws for all 2 draws for all 1.420 (0.424) 0.979 (0.573) 0.919 (0.481) 2.911 (2.776) 0.798 (0.532) 0.258 (0.212) 3 3 draws for all 1 draws for all 1.668 (0.554) 1.349 (0.348) 1.033 (0.612) 2.531 (1.920) 0.718 (0.817) 0.366 (0.455) 4 3 draws for all 1 draws for half 1.596 (0.603) 0.851 (0.324) 1.072 (0.604) 0.951 (1.049) 0.798 (0.580) 0.704 (0.691) 5 3 draws for all Two groups of public info 1.528 (0.600) 0.798 (0.451) 1.174 (0.652) 0.886 (0.763) 1.015 (0.751) 0.472 (0.397) KL Measures for Public Info Experiments Experimental Economics Program

  39. Summary • IAM with public info correction did better than best person. • IAM with public info correction did better than markets in 4 out of 5 cases. • IAM corrected with true public info did significant better than all other methods. Experimental Economics Program

  40. Experimental Economics Program

  41. Experimental Economics Program

  42. Supplementary Experimental Economics Program

  43. Previous Research • Academic Studies • Information Aggregation in Markets • Plott, Sunder, Camerer, Forsythe, Lundholm, Weber,… • Pari-mutuel Betting Markets • Plott, Wit & Yang • Real World Applications • Iowa Electronic Markets • Hollywood Stock Exchange • HP Information Markets • Newsfuture • Tradesport.com • … Experimental Economics Program

  44. Risk Attitudes Experimental Economics Program

  45. Dealing with Risks Attitudes:Two-Stage Mechanism Stage 1: Information Market Call Market to Solicit Risk Attitudes Time Event 1 Event 2 Event 3 Event 4 Stage 2: Probability Reporting & Aggregation Individual Report of Probability Distribution Nonlinear Aggregated Function Event 5 Event 6 Event 7 Event 8 Experimental Economics Program

  46. Probability Reporting Outcome A B C Reports of Probability Distribution p1 p2 p3 Pays C1+C2*Log(p3) Experimental Economics Program

  47. Second Stage: Aggregation Function Bayes Law with Behavioral Correction Normalizing constant for individual risks i=r(V i / i)c Holding value/Risk - measure relative risk of individuals “market” risk ~sum of prices/winning payoff Experimental Economics Program

  48. Private Information Experiments • 5 Experiments • Various Information Conditions • All subject received 3 draws • Half received 5 draws, half received 1 draw • Half received 3 draws, half received random number of draws • 8 to 13 participants in each experiments Experimental Economics Program

  49. Next Step • Field Test (Fine and Huberman) … Experimental Economics Program

More Related