1 / 61

Crowdsourcing and its applications on Scientific Research

Explore the applications and impact of crowdsourcing in scientific research, including examples of tasks, platforms like Amazon Mechanical Turk, and the characteristics of crowd workers.

jasonwalker
Download Presentation

Crowdsourcing and its applications on Scientific Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Crowdsourcing and its applications on Scientific Research

  2. Crowdsourcing = Crowd + Outsourcing • “soliciting solutions via open calls to large-scale communities”

  3. Some Examples • Call for professional helps • Award 50,000 to 1,000,000 for each tasks • Office work platform • Microtask platform • Over 30,000 tasks at the same time

  4. What Tasks are crowdsourceable?

  5. Software Development • Reward: 25,000 USD

  6. Data Entry • Reward: 4.4 USD/hour

  7. Image Tagging • Reward: 0.04 USD

  8. Trip Advice • Reward: points on Yahoo! Answers

  9. The impact of crowdsourcingon scientific research?

  10. Amazon Mechanical Turk • A micro-task marketplace • Task prices are usually between 0.01 to 1 USD • Easy-to-use interface

  11. Amazon Mechanical Turk • Human Intelligence Task (HIT) • Tasks hard for computers • Developer • Prepay the money • Publish HITs • Get results • Worker • Complete the HITs • Get paid

  12. Who are the workers?

  13. A Survey of Mechanical Turk • Survey on 1000 Turkers (Turk workers) • Two identical surveys (Oct. 2008 and Dec. 2008) • Consistent results • Blog post: • A Computer Scientist in a Business School

  14. Age Gender Education Annual Income

  15. Compare with Internet Demographics • Use the data from ComScore • In summary, Tukers are • younger • Portion of 21-35 years old: 51% vs. 22% in internet • mainly female • 70% female vs. 50 % female • having lower income • 65% turkers with income < 60k/year vs. 45% in internet • having smaller family • 55% turkers have no children vs. 40% in internet

  16. How Much Turkers Earn?

  17. Why Turkers Turk?

  18. Research Applications

  19. Dataset Collection • Dataset is important in computer science! • In multimedia analysis • Is there X in the image • Where is Y in the image • In natural language processing • What is the emotion of this sentence • And in lots of other applications

  20. Dataset Collection • Utility Annotation • By Sorokin and Forsyth at UIUC • Image analysis • Type keyword • Select examples • Click on landmarks • Outline figures

  21. 0.01 USD/ task

  22. 0.02 USD/ task

  23. 0.01 USD/ task

  24. 0.01 USD/ task

  25. Dataset Collection • Linguistic annotations (Snow et al. 2008) • Word similarity USD 0.2 to label 30 word pairs

  26. Dataset Collection • Linguistic annotations (Snow et al. 2008) • Affect recognition USD 0.4 to label 20 headlines (140 labels)

  27. Dataset Collection • Linguistic annotations (Snow et al. 2008) • Textual entailment • If “Microsoft was established in Italy in 1985”, then “Microsoft was established in 1985” ? • Word sense disambiguation • “a bass on the line” vs. “a funky bass line” • Temporal annotation • Ran happens before fell: • “The horse ran past the barn fekk”

  28. Dataset Collection • Document relevance evaluation • Alonso et al. (2008) • User rating collection • Kittur et al. (2008) • Noun compound paraphrasing • Nakov (2008) • Name resoluation • Su et al. (2007)

  29. Data Characteristic Cost? Efficiency? Quality?

  30. Cost and Efficiency • In image annotation • Sorokin and Forsyth, 2008

  31. Cost and Efficiency • In linguistic annotation • Snow et. al, 2008

  32. Cheap and fast! Is it good?

  33. Quality • Multiple non-experts can beat experts • 三個臭皮匠勝過一個諸葛亮 • Black line • agreement among turkers • Green line: • single expert • Golden result: • agreement among multiple experts

  34. In addition to Dataset Collection

  35. QoE Measurement • QoE (Quality of Experience) • Subjective measure of user perception • Traditional approach • User studies by MOS ratings (Bad -> Excellent) • Crowdsourcing with paired comparison • Diverse user input • Easy to understand • Interval scale scores can be calculated

  36. Acoustic QoE Evaluation

  37. Acoustic QoE Evaluation • Which one is better? • Simple pair comparison

  38. Optical QoE evaluation

  39. Interactive QoE Evaluation

  40. Acoustic QoE • VoIP Loss Rate • MP3 Compression Rate

  41. Optical QoE • Packet loss rate • Video Codec

  42. Iterative Task

  43. Iterative Tasks • Turkit: tools for iterative tasks on Mturk • Imperative programming paradigm • Basic elements • Variable (a = b) • Control (if else statement) • Loop (for, while statement) • Turning MTurk into a programming platform which integrates human brain powers

  44. Iterative Text Improvement • A Wikipedia-like scenario • One Turkerimprove the text • Other Turkersvote if the improvement is valid

  45. Iterative Text Improvement • Image description • Instructions for the improve-HIT • Please improve the description for this image • People will vote whether to approve your changes • Use no more than 500 characters • Instructions for the vote-HIT • Please select the better description for this image • Your vote must agree with the majority to be approved

  46. Iterative Text Improvement • Image description • A partial view of a pocket calculator together with some coins and a pen. • A view of personal items a calculator, and some gold and copper coins, and a round tip pen, these are all pocket and wallet sized item used for business, writing, calculating prices or solving math problems and purchasing items. • A close-up photograph of the following items:* A CASIO multi-function calculator* A ball point pen, uncapped* Various coins, apparently European, both copper and gold • …Various British coins; two of £1 value, three of 20p value and one of 1p value. …

  47. Iterative Text Improvement • Image description A close-up photograph of the following items: A CASIO multi-function, solar powered scientific calculator. A blue ball point pen with a blue rubber grip and the tip extended. Six British coins; two of £1 value, three of 20p value and one of 1p value. Seems to be a theme illustration for a brochure or document cover treating finance - probably personal finance.

  48. Iterative Text Improvement • Handwriting Recognition • Version 1 • You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few grammatical mistakes. Overall your writing style is a bit too (phoney). You do (?) have good (points), but they got lost amidst the (writing). (signature)

More Related