Using Informative Priors to Enhance Wisdom in Small Crowds

1. Using Informative Priors to Enhance Wisdom in Small Crowds

2. Eyewitness testimony • Common problems • Reliance on error-prone eyewitnesses • Reliance on very small number of eyewitnesses (often one) • This research • Integrating recalled memories across multiple individuals • Focus on situations involving small number of individuals with potentially poor memory • Approach • Incorporate informative priors into a wisdom of crowd aggregation model • Prior knowledge is extracted from a different group of individuals

3. Illustrative example: Galton’s Ox • Galton’s original experiment • estimate weight of ox (true answer is 1198 pounds) • large number of individuals (800) • median answer came within 9 pounds of true weight • Thought experiment • Suppose we only have small numbers of individuals • Suppose we know something about ox weight a priori

4. Galton’s Ox with Prior Information j=1,..,N Goal: infer the true mean from the observed estimates Assumptions:

5. Bayesian Inference • Standard solution: • Measure mean absolute difference

6. Simulation • Vary the number of individuals (N) and strength of the assumed prior by the researcher (σ0) uninformative prior (based on 50,000 repetitions of Galton’s experiment in each cell)

7. Application to Episodic Memory • We apply the same ideas to an episodic memory task: serial recall • Serial recall experiment (N=28) • Study event sequence in video • Order the test images from memory • Norming experiment (N=16) • No study phase • Order test images in as natural order as possible • Allows us to build a model for prior probability of each sequence

8. Materials • Type I • 3 videos with stereotyped event sequences (e.g. wedding) • associated with “strong prior” • Type II • 3 videos with less predictable event sequence • associated with “weak prior” • Extracted 10 images for testing

9. Example Type I Sequence: Bus video prior serial recall Note: first row is subject id, second row is Kendall tau to true ordering

10. Example Type II Sequence: Clay Video prior serial recall

11. Empirical Results: distribution of tau distances Type I Type II Note: bin width is two so first bar is the relative proportion to tau = 0 and tau = 1

12. Mallows Model Kendall tau distance Latent truth Recalled order Scaling parameter Normalizing constant

13. Model 1: Mallows Model with Uninformative Prior ω yj θj j=1,..,M

14. Model 2: Mallows Model with Informative Prior θ* θoj yoj ωo ω yj θj j=1,..,N j=1,..,M (prior knowledge data) (memory data)

15. Example calibration result Model 1 Model 2 (clay video; all subs)

16. Wisdom of Crowds Effect (Model 1)

17. Pick worst K individuals (from episodic task) (means of taus across mcmc samples; averaged across 3 videos for each type)

18. Modeling Results(pick worst K individuals; means of taus across mcmc samples)

19. Pick random K individuals (means of taus across mcmc samples; averaged across 3 videos for each type)

20. Pick random K individuals means of taus across mcmc samples

21. Left over

22. Pick worst K individuals (single tau)

23. Modeling Results(pick worst K individuals; single tau – use Borda count to aggregate samples)

24. This research • Wisdom of crowds • Recover original sequence of events based on recollected order from a group of individuals • Prior knowledge • We measure prior knowledge about event sequences • Incorporate this prior knowledge into our wisdom of crowd aggregation model  informative priors