1 / 15

CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility

CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility. Alex Quinn, Ben Bederson, Tom Yeh , Jimmy Lin. Human Computation. Translation Photo tagging Face recognition Human detection Speech recognition Text analysis Planning. Things COMPUTERS can do.

montana
Download Presentation

CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CrowdFlowIntegrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin

  2. Human Computation Translation Photo tagging Face recognition Human detection Speech recognition Text analysis Planning ThingsCOMPUTERScan do ThingsHUMANScan do

  3. Human Computation Translation Photo tagging Face recognition Speech recognition Human detection Text analysis Planning ThingsCOMPUTERScan do ThingsHUMANScan do

  4. Example: Human detection

  5. Trade-off space Computers Speed, Affordability Human Computation Human Workers(traditional) Quality

  6. Trade-off space Computers Speed, Affordability Human Computation Human Workers(traditional) Quality

  7. Man-Computer Symbiosis humans computer computer humans speed cost quality speed cost quality Supervisedmachinelearning Automation with human post-correction

  8. Man-Computer Symbiosis humans computer computer humans computer humans speed cost quality speed cost quality speed cost quality Supervisedmachinelearning Automation with human post-correction CrowdFlow

  9. Mechanical Turk

  10. Human Detection – Starting point

  11. Human Detection – Task

  12. Human Detection – Results Speed, Affordability 119 images took3 hrs 50 mins and cost $2.38 Quality 60% 90%

  13. Human Detection – Scenarios 1000 photos at 72% accuracy would take 12 hrs 20 mins and cost $8.00 Speed, Affordability 119 images took3 hrs 50 mins and cost $2.38 Quality 60% 90%

  14. Vision: Richer model Input with computer results Correct Incorrect Validator Start over Fix Appraiser Worker Fixer Output

  15. Lessons Learned • Design for overall needs/constraints • Practical advice: • Pay consistently and reasonably • Reject only work that is definitely cheating • Build in fair cheating deterrence from the start • Keep instructions short, but always clear Contact: Alex Quinn aq@cs.umd.edu

More Related