Easy (and cheap) data Diverse Intuition pump

6. (My) reasons for using crowdsourcing Easy (and cheap) data Diverse Intuition pump

7. (My) reasons for using crowdsourcing Easy (and cheap) data Diverse Intuition pump

8. Easy (and cheap) data Median hit - $0.06 Iperotis, 2010b

9. (My) reasons for using crowdsourcing Easy (and cheap) data Diverse Intuition pump

10. >500,000 “Workers” ~35-40% US; 30-50% India + ~100 countries

11. Heinrich, Heine, Norenzayan, 2010 Western Educated Rich Industrialized Democracies “a randomly selected American undergraduate is more than 4,000 times more likely to be a research participant than is a randomly selected person from outside of the West” (p. 63)

12. (My) reasons for using crowdsourcing Easy (and cheap) data Diverse Intuition pump

13. N=126 (135 tested) $2.70 + 10%

15. How acceptable is it to say: He speeded down the road He lighted the candles They sneaked around etc. r = :23, p = :025 Logit reg. odds 􀀀 ratio = :78, p = :028 Dale & Lupyan, 2011

16. Whoworks Self reported demographic information from 2,896 workers over 3 years (MW ‘09, MW ‘11, SW ’10) 55% Female, 45% Male • Similar to other internet panels (e.g. Goldstein) Age: • Mean: 30 yrs, • Median: 32 yrs Mean Income: $30,000 / yr Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

17. Why “MTurkmoney is always necessary to make ends meet.” • 5% U.S. 13% India “MTurkmoney is irrelevant.” • 12% U.S. 10% India “MTurkis a fruitful way to spend free time and get some cash.” • 69% U.S. 59% India Ross et al ’10, Ipeirotis ’10 Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

18. Common Tasks What Image labeling Audio transcription Classification: images, websites Product evaluation Uncommon Tasks Workflow optimization Copy editing Product description Technical writing Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

19. Whorequests CrowdFlower is the leader in enterprise crowdsourcing. CrowdFlower’s technology platform offers quality-ensured crowdsourcing at massive scale. The company solves problems ranging from product categorization to business lead verification to content creation. Clients from startups to the Fortune 500 enjoy increased flexibility, faster turnaround time and cost savings. …..CrowdFlowertakes large, data-heavy projects and breaks them into small tasks that are distributed to more than 1.5 million on-demand contributors globally. Companies crowdsourcing part of their business Search companies: relevance Online stores: similar products from different stores (identifying competition) Online directories: accuracy, freshness of listings Researchers Intermediaries CrowdFlower Smartsheet.com Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

20. Whorequests What Iperotis, 2010b

21. Anatomy of a HIT

22. Anatomy of a HIT • HITs with the same title, description, pay rate, etc. are the same HIT type • (for us, it’s mostly 1 HIT / Hit Type) • HITs are broken up into Assignments • A worker cannot do more than 1 assignment of a HIT • Not everyone is eligible to do every HIT Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

23. Some HIT groups have many HITs Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

24. HIT GROUP Assignment 1 “Black” Alice Assignment 2 “Night” • Which is the better translation for Táy? • Black • Night HIT 1 Bob • Which is the better translation for Nedj? • Clean • White HIT 2 Assignment 3 “Black” Charlie • • • Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

25. HIT GROUP Assignment 1 “White” • Which is the better translation for Táy? • Black • Night Alice HIT 1 Assignment 2 “White” • Which is the better translation for Nedj? • Clean • White HIT 2 Bob • • Assignment 3 “White” • David Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

26. Requester Worker Build HIT Test HIT Search for HITs Post HIT Accept HIT Do work Reject or Approve HIT Submit HIT Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

27. How much $? Accuracy Number of Tasks Completed Pay per Task Pay per Task Pay rate can affect quantity of work Pay rate does not have a big impact on quality Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

28. Completion Time 3, 6-question multiple choice surveys Launched same time of day, day of week $0.01, $0.03, $0.05 Past a threshold, pay rate does not increase speed Start with low pay rate work up Slide adapted from Winter Mason’s presentation: http://www.slideshare.net/winteram

29. Turker Community Reputation of Workers is given by approval rating Requesters can reject work Requesters can refuse workers with low approval rates Workers can also rate requesters (stay tuned)

31. Validity

32. How

33. Qualtrics and Mturk -As good as point and click gets

35. Imagine a shape called a "foove“ (“crelch”) Would such a shape be...

40. Imagine that the word ‘crelch’refers to a shape. Draw what you think the shape looks like Imagine that the word‘foove’refers to a shape. Draw what you think the shape looks like

41. If you can code it, you can run it.

44. Test run #1 • Recruited through social network contacts. • Overall, about ~60 players, produced 1,377 squiggles, and “listened” 4,136 times. • Collected in 1 day. • One user played for 1 hour straight.

45. Test Run 2 • 53 participants from Amazon’s crowdsourcing service Mechanical Turk • 1,034 squiggles, > 3,000 listens • Collected in under 1 day • Some users played for over 30 minutes. • Effective hourly rate: • $0.15 / hour

46. Tips Be creative. Don’t try to force a lab study into a Turk format.

47. Thibodeau & Boroditsky, 2011

49. Tips Validate the work when necessary.