1 / 21

Utility data annotation via Amazon Mechanical Turk

Utility data annotation via Amazon Mechanical Turk. Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign http://visionpc.cs.uiuc.edu/~largescale/. X 100 000 = $5000 . Motivation. Unlabeled data is free Labels are useful

lindsey
Download Presentation

Utility data annotation via Amazon Mechanical Turk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Utility data annotation via Amazon Mechanical Turk Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign http://visionpc.cs.uiuc.edu/~largescale/ X 100 000 = $5000

  2. Motivation • Unlabeled data is free • Labels are useful • We need large volumes of labeled data • Different labeling needs: • Is there Xin the image? • Outline X. • Where is part Y of X. • Of these 500 images, which belong to category X? • ……………. and many more ……………….

  3. Amazon Mechanical Turk Workers Task Task: Dog? Broker Answer: Yes Pay: $0.01 Is this a dog? www.mturk.com o Yes o No $0.01

  4. Motivation X 100 000 = $5000 Custom annotations Large scale Low price

  5. Annotation protocols • Type keywords • Select relevant images • Click on landmarks • Outline something • Detect features ……….. anything else ………

  6. Type keywords $0.01 http://austinsmoke.com/turk/.

  7. Select examples Joint work with Tamara and Alex Berg http://visionpc.cs.uiuc.edu/~largescale/data/simpleevaluation/html/horse.html

  8. Select examples $0.02 requester mtlabel

  9. Click on landmarks $0.01 http://vision-app1.cs.uiuc.edu/mt/results/people14-batch11/p7/

  10. Outline something $0.01 http://visionpc.cs.uiuc.edu/~largescale/results/production-3-2/results_page_013.html Data from Ramanan NIPS06

  11. Detect features Measuring molecules. Joint work with Rebecca Schulman (Caltech) ?? $0.1 http://visionpc.cs.uiuc.edu/~largescale/all_examples.html

  12. Motivation X 100 000 = $5000 Custom annotations Large scale Low price

  13. Issues • Quality? • How good is it? • How to be sure? • Price? • How to price it? • How does MTurk compare with others? • How do I sign up? sorokin2@uiuc.edu http://visionpc.cs.uiuc.edu/~largescale/

  14. Annotation quality Agree within 5-10 pixels on 500x500 screen There are bad ones. A C E G

  15. Grading tasks • Take 10 submitted results • Create new task to verify the result • Verification is easy • Pay the same or slightly higher price • Total overhead - 10% (work in progress) http://vision-app1.cs.uiuc.edu/mt/grading/people14-batch11-small/p1/

  16. Price • $0.01 per image (16 clicks) ~ $1500 / 100 000 images >1000 images per day <4 months • Workers suggested $0.03 - $0.05/img • $3500 - $5500 / 100 000 images

  17. Is the price right? $0.01/ 40 clicks 15 hours 900 labels $0.01 / 14 clicks 1.6 hours 900 labels $0.01 / 16 clicks 4 hours 900 labels

  18. Annotation Method Comparison

  19. How do I sign up? • Go to our web page: http://visionpc.cs.uiuc.edu/~largescale/ • Send us an e-mail: sorokin2@uiuc.edu • Register at Amazon Mechanical Turk http://www.mturk.com

  20. Acknowledgments Special thanks to: David Forsyth Tamara Berg Rebecca Schulman David Martin Kobus Barnard Mert Dikmen All workers at Amazon Mechanical Turk This work was funded in part by ONR

  21. Thank you X 100 000 = $5000

More Related