Crowdsourcing Control: Moving Beyond Multiple Choice (Full paper in UAI 2012)

1. Crowdsourcing Control: Moving Beyond Multiple Choice(Full paper in UAI 2012) Christopher H. Lin, Mausam, Daniel S. Weld University of Washington HCOMP-2012, Toronto, ON, Canada TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

2. www.crowdsourcing.com Yes No Is this a face? Yes No

3. www.crowdsourcing.com Yes No Is this a face? Yes No

4. www.crowdsourcing.com Yes No Is this a face? Yes No

5. www.crowdsourcing.com Yes No Is this a face? Yes No

6. Update Posteriors

7. www.crowdsourcing.com How many people are there in this military base? 30,000 Submit Source:Wikipedia

8. lazysusan Update Posteriors

9. lazysusan Update Posteriors

10. www.crowdsourcing.com 2 + 2 = ?

11. www.crowdsourcing.com EASY 2 + 2 = ? d = 0

12. www.crowdsourcing.com ? =

13. www.crowdsourcing.com ? HARD = d = 1

14. Source:Wikipedia

15. Accuracy Worker Ability Difficulty

16. [Aldous, 1985]

17. A C B ??

18. A C B ??

19. + A C B ??

20. A C B ??

21. + A C B ??

22. Agent State Difficulty Correct Answer

24. Learn New Parameters

29. Contributions • A probabilistic generative model for worker responses on tasks with an infinite number of possible answers • A decision-theoretic agent that dynamically infers the right answer • Real experiments that show our agent works

30. Repeated Labeling Romney et al, 1986; Sheng et al, 2008; Raykar et al, 2010; Whitehill et al, 2009; Dai et al, 2010, 2011; Kamar et al, 2012; Parameswaran et al, 2010; Karger et al, 2011; Snow et al, 2008; Complicated Workflows Little et al, 2009; Bernstein et al, 2010; Liem et al, 2011; Noronha et al, 2011; Lasecki et al, 2011; Kulkarni et al, 2012; AI Control of Workflows Dai et al, 2010,2011; Shahaf et al, 2010; Weld et al, 2011; Kamar et al, 2012;

31. Christopher H. Lin, Mausam, and Daniel S. Weld. “Dynamically Switching between Synergistic Workflows.” Wednesday, AI and the Web 2 2:25 PM