Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals

1. Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Brent Miller, Pernille Hemmer, Mike Yi Michael Lee, Bill Batchelder, Paolo Napoletano

2. What is the correct chronological order? Abraham Lincoln Ulysses S. Grant time Ulysses S. Grant Rutherford B. Hayes Rutherford B. Hayes James Garfield Abraham Lincoln Andrew Johnson James Garfield Andrew Johnson

3. Research goal: aggregating responses ground truth group answer ? A B C D = A B C D Aggregation Algorithm A D B C D A B C B A D C A C B D A B D C

4. Task constraints • No communication between individuals • There is always a true answer (ground truth) • Aggregation algorithm is unsupervised • ground truth only used for evaluation

5. Wisdom of crowds phenomenon • Group estimate often performs as well as or better than best individual in the group

6. Examples of wisdom of crowds phenomenon Galton’s Ox (1907): Median of individual estimates comes close to true answer Who wants to be a millionaire?

7. Relation to Cultural Consensus Theory (CCT) • Developed by Batchelder and Romney • CCT can recover the answer key of a multiple choice test by analyzing responses across individuals • Key assumption: questions vary in difficulty and individuals vary in ability • Our models will be similar to the ideas of CCT, but the emphasis is different • Each problem studied has a ground truth • We focus on “wisdom of crowds” phenomenon

8. Overview of talk • Ordering problems – general knowledge • what is the order of US presidents? • Ordering problems – episodic memory • what is the order of events you have experienced? • Matching problems • memory for pairs: what object was paired with what person? • Recognition memory problems • what words were studied?

9. Experiment: 26 individuals order all 44 US presidents

10. Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps = 1 = 1+1 = 2 Ordering by Individual A B E C D A B E CD E C D A B C D E A B True Order A B C D E

11. Empirical Results (random guessing) t

12. A Bayesian (generative) approach latent “input” ? ? ? ? Model Model Model Model … D A B C B A D C A C B D A B D C

13. Bayesian models • We extend two models: • Thurstone’s (1927) model • Estes (1972) perturbation model

14. Bayesian Thurstonian Approach C B A Each item has a true coordinate on some dimension

15. Bayesian Thurstonian Approach Person 1 B A C … but there is noise because of encoding and/or retrieval error

16. Bayesian Thurstonian Approach Person 1 B A C B C A Each person’s mental representation is based on (latent) samples of these distributions

17. Bayesian Thurstonian Approach Person 1 B A C Observed Ordering: A < B < C B C A The observed ordering is based on the ordering of the samples

18. Bayesian Thurstonian Approach Person 1 B A C Observed Ordering: A < B < C B C A Person 2 B C A Observed Ordering: A < C < B C B A People draw from distributions with common means but different variances

19. Graphical Model Notation j=1..3 shaded = observed not shaded = latent

20. Graphical Model of Bayesian Thurstonian Model Latent group means Individual noise level Mental representation Observed ordering j individuals

21. Inference • Need the posterior distribution • Markov Chain Monte Carlo • Gibbs sampling on • Metropolis-hastings on and

22. Inferred Distributions for 44 US Presidents George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) median and minimumsigma James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44)

23. Model can predict individual performance t individual t distance to ground truth s inferred noise level for each individual

24. (Weak) Wisdom of Crowds Effect t model’s ordering is as good as best individual (but not better)

25. Extension of Estes (1972) Perturbation Model • Main idea: • item order is perturbed locally • Our extension: • perturbation noise varies between individuals and items True order A B C D E A C B D E Recalled order

26. Modified Perturbation Model

27. Inferred Perturbation Matrix and Item Accuracy Abraham Lincoln Richard Nixon James Carter

28. Strong wisdom of crowds effect t Perturbation Perturbation model’s ordering is better than best individual

29. Alternative Heuristic Models • Many heuristic methods from voting theory • E.g., Borda count method • Suppose we have 10 items • assign a count of 10 to first item, 9 for second item, etc • add counts over individuals • order items by the Borda count • i.e., rank by average rank across people

30. Model Comparison t Borda

31. Ordering Ten Amendments Freedom of speech & religion (1) Right to bear arms (2) No quartering of soldiers (4) No unreasonable searches (3) Due process (5) Trial by Jury (6) Civil Trial by Jury (7) No cruel punishment (8) Right to non-specified rights (10) Power for the States & People (9)

32. Ordering Ten Commandments

33. Overview of talk • Ordering problems – general knowledge • what is the order of US presidents? • Ordering problems – episodic memory • what is the order of events you have experienced? • Matching problems • memory for pairs: what object was paired with what person? • Recognition memory problems • what words were studied?

34. Recollecting order from episodic memory http://www.youtube.com/watch?v=a6tSyDHXViM&feature=related

35. Place scenes in correct order (serial recall) A B C D time

36. Recollecting Order from Episodic Memory Study this sequence of images

37. Place the images in correct sequence (serial recall) A B C D E F G H I J

38. Average results across 6 problems t Mean

39. Example calibration result for individuals t individual distance to ground truth s inferred noise level (pizza sequence; perturbation model)

40. Overview of talk • Ordering problems – general knowledge • what is the order of US presidents? • Ordering problems – episodic memory • what is the order of events you have experienced? • Matching problems • memory for pairs: what object was paired with what person? • Recognition memory problems • what words were studied?

41. Study these combinations

42. Find all matching pairs C A B D E 1 2 3 4 5

43. Bayesian Matching Model • Proposed process: • match “known” items • guess between remaining ones • Individual differences • some items easier to know • some participants know more

44. Graphical Model person ability item easiness iitems Prob. of knowing Latent answer key Knowledge State Observed matching j individuals

45. Results across 8 problems

46. General Knowledge Matching Problems

47. Modeling Results – General Knowledge Tasks

48. Overview of talk • Ordering problems – general knowledge • what is the order of US presidents? • Ordering problems – episodic memory • what is the order of events you have experienced? • Matching problems • memory for pairs: what object was paired with what person? • Recognition memory problems • what words were studied?

49. Systematic Errors and Biases • Some memory errors are systematic • When averaging over biased individuals, the group estimate will also be systematically biased … unless the aggregation model can explain the bias

50. Listen to these words…