Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals

1. Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Brent Miller, Pernille Hemmer, Mike Yi Michael Lee, Bill Batchelder, Paolo Napoletano

2. Wisdom of crowds phenomenon • Group estimate often performs as well as or better than best individual in the group

3. Examples of wisdom of crowds phenomenon Galton’s Ox (1907): Median of individual estimates comes close to true answer Who wants to be a millionaire?

4. Recollection of 9/11 Event Sequence (Altmann, 2003) Most frequent response (i.e, mode) A = One plane hits the WTC B = A second plane hits the WTC C = One plane crashes into the Pentagon D = One tower at the WTC collapses E = One plane crashes in Pennsylvania F = A second tower at the WTC collapses Correct

5. Research goal: aggregating responses ground truth group answer ? A B C D = A B C D Aggregation Algorithm A D B C D A B C B A D C A C B D A B D C

6. Task constraints • No communication between individuals • There is always a true answer (ground truth) • Aggregation algorithm never has access to ground truth • unsupervised methods • ground truth only used for evaluation

7. Is this research part of psychology? • Yes • Effective aggregation of human judgments requires cognitive models

8. Overview of talk • Ordering problems • what is the order of US presidents? • Matching problems • memory for pairs: what object was paired with what person? • Recognition memory problems • what words were studied? • Experts in crowds • how to find experts in the absence of feedback

9. Recollecting Order from Declarative Memory Abraham Lincoln Ulysses S. Grant time Ulysses S. Grant Rutherford B. Hayes Rutherford B. Hayes James Garfield Abraham Lincoln Andrew Johnson James Garfield Andrew Johnson Place these presidents in the correct order

10. Experiment: Order all 44 US presidents • Similar to Roediger and Crowder (1976); Healy, Havas, Parker (2000) • Methods • 26 participants (college undergraduates) • Names of presidents written on cards • Cards could be shuffled on large table

11. Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps = 1 = 1+1 = 2 Ordering by Individual A B E C D A B E CD E C D A B C D E A B True Order A B C D E

12. Empirical Results (random guessing) t

13. A Bayesian (generative) approach shared group knowledge A B C D (latent random variable) Generative Model A D B C D A B C B A D C A C B D A B D C

14. Bayesian models • We extend two models: • Thurstone’s(1927) model • Estes (1972) perturbation model

15. Bayesian Thurstonian Approach C B A Each item has a true coordinate on some dimension

16. Bayesian Thurstonian Approach Person 1 B A C … but there is noise because of encoding and/or retrieval error

17. Bayesian Thurstonian Approach Person 1 B A C B C A Each person’s mental representation is based on (latent) samples of these distributions

18. Bayesian Thurstonian Approach Person 1 B A C Observed Ordering: A < B < C B C A The observed ordering is based on the ordering of the samples

19. Bayesian Thurstonian Approach Person 1 B A C Observed Ordering: A < B < C B C A Person 2 B C A Observed Ordering: A < C < B C B A People draw from distributions with common means but different variances

20. Bayesian Inference Problem • Given the orderings from individuals, infer: • mean for each item • standard deviations for each person • Markov Chain Monte Carlo (MCMC)

21. Inferred Distributions for 44 US Presidents George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) median and minimumsigma James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44)

22. Model can predict individual performance t individual t distance to ground truth s inferred noise level for each individual

23. (Weak) Wisdom of Crowds Effect t model’s ordering is as good as best individual (but not better)

24. Extension of Estes (1972) Perturbation Model • Main idea: • item order is perturbed locally • Our extension: • perturbation noise varies between individuals and items True order A B C D E A C B D E Recalled order

25. Inferred Perturbation Matrix and Item Accuracy Abraham Lincoln Richard Nixon James Carter

26. Strong wisdom of crowds effect t Perturbation Perturbation model’s ordering is better than best individual

27. Alternative Heuristic Models • Many heuristic methods from voting theory • E.g., Borda count method • Suppose we have 10 items • assign a count of 10 to first item, 9 for second item, etc • add counts over individuals • order items by the Borda count • i.e., rank by average rank across people

28. Model Comparison t Borda

29. Recollecting order from episodic memory http://www.youtube.com/watch?v=a6tSyDHXViM&feature=related

30. Place scenes in correct order (serial recall) A B C D time

31. Recollecting Order from Episodic Memory Study this sequence of images

32. Place the images in correct sequence (serial recall) A B C D E F G H I J

33. Average results across 6 problems t Mean

34. Example calibration result for individuals t individual distance to ground truth s inferred noise level (pizza sequence; perturbation model)

35. Overview of talk • Ordering problems • what is the order of US presidents? • Matching problems • memory for pairs: what object was paired with what person? • Recognition memory problems • what words were studied? • Experts in crowds • how to find experts in the absence of feedback

36. Study these combinations

37. Find all matching pairs C A B D E 1 2 3 4 5

38. Results across 8 problems

39. General Knowledge Matching Problems

40. Modeling Results – Declarative Tasks

41. Overview of talk • Ordering problems • what is the order of US presidents? • Matching problems • memory for pairs: what object was paired with what person? • Recognition memory problems • what words were studied? • Experts in crowds • how to find experts in the absence of feedback

42. Listen to these words…

43. Experiment • Study list • 10 lists of 15 spoken words • Recognition memory test • Targets (15 items) • Lure (1 item) • Related distractors (15 items) • Unrelated distractors (15 items) • Confidence ratings • 5-point confidence ratings • 1=definitely not on list; 2 = probably not on list; 3 = not sure; 4 = probably on list; 5 = sure it was on the list

44. Mean confidence ratings for 12 subjects Confidence

45. ROC plots for individuals

46. Heuristic Aggregation Method • Group confidence = mean confidence rating across individuals

47. Performance of Aggregate

48. Performance of Individuals and Aggregate

49. Problem with aggregation method • Aggregate also suffers from false memories Confidence

50. Potential Solution: identify group signature of false memories