1 / 32

Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs

Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs. Alex Quinn, Ben Bederson. R. L. Polk & Co. / Autotrader.com, Automotive Buyer Study, 2011.

brick
Download Presentation

Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson

  2. R. L. Polk & Co. / Autotrader.com, Automotive Buyer Study, 2011 “Market research firm J.D. Power and Associates says […] more than 80% of buyers have already spent an average of 18 hours online researching models and prices, according to Google data.” Wall Street Journal, 2/26/2013

  3. DATA-DRIVEN DECISIONS

  4. Building blocks for Mechanical Turk:HITs (human intelligence task)

  5. Keep instructions short. Input labels should be unambiguous HITs must be grouped by common templates See Mechanical Turk Requester Best Practices Guide

  6. Example #1: Find a pediatrician • Requirements • Accepts my insurance • ≥4 stars at RateMDs.com • >80% positive at HealthGrades.com • ≤15 minutes drive from home

  7. Effort should be proportional to the reward. HITs in a group share a common base price. Information sources should be traceable. See Mechanical Turk Requester Best Practices Guide

  8. Example #2: Buy a stroller • Requirements • Fits a 30-pound baby • Reclines for sleeping • Medium/large-sized soft tires • Can purchase online in US

  9. Find creative ways to track sources Bonus offers allow reward to scale with effort

  10. Design lessons • Consider effort-reward balance from the start. • Look for implicit ways of capturing sources. • Use word economy to conserve vertical space. • Choose unambiguous input labels. Alex Quinn aq@cs.umd.edu

More Related