1 / 24

Logging the Search Self-Efficacy of Amazon Mechanical Turkers

This study explores the relationship between search self-efficacy, search frustration, and the effectiveness of search assistance tools among Amazon Mechanical Turks. Initial experiments were conducted to understand the spread of search self-efficacy and the impact of price on speed and spread. Challenges include scale modifications, data reliability, and ethical considerations.

kcharlton
Download Presentation

Logging the Search Self-Efficacy of Amazon Mechanical Turkers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Logging the Search Self-Efficacy of Amazon Mechanical Turkers Henry Feild* (UMass) Rosie Jones* (Akamai) Robert Miller (MIT) Rajeev Nayak (MIT) Elizabeth Churchill (Yahoo!) EmreVelipasaoglu (Yahoo!) July 23, 2010 * Work done while at Yahoo!

  2. Imagine you are frustrated searching and think you are a good searcher... ✗✔ ✗✔ ✗✔ ✗✔

  3. Imagine you are frustrated searching andthinkyou area goodsearcher... bad ✗✔ ✗✔ ✗✔ ✗✔

  4. Outline • What we’re trying to do • Search self-efficacy • Searcher frustration • Search Assistance • AMT – why use it? • Initial experiments • Challenges

  5. What we’re trying to do • What relationships exist between: • a user’s search self-efficacy, • their current level of search frustration, • and what search assistance they find most helpful

  6. Search self-efficacy • how good of a searcher one perceives themselves to be • measured using a scale • related work: Diane Kelly[Tech report, 2010] • I can... • Find articles similar in quality to those obtained by a professional searcher. • Devise a query which will result in a very small percentage of irrelevant items on my list. • ...

  7. Searcher Frustration • how frustrated a user is while searching for an information need • measured using a scale • example: • What was the best selling TV model in 2008? • television set sales 2008 • “television set” sales 2008 • “television” sales 2008 • google trends • “television” sales statistics 2008 user got frustrated starting here

  8. Search Assistance • a tool that assists with search • examples: • suggest as you type • query suggestions • relevance feedback ✗✔

  9. Study platform

  10. Outline • What we’re trying to do • Search self-efficacy • Searcher frustration • Search Assistance • AMT – why use it? • Initial experiments • Challenges

  11. AMT – Why use it? • we can cover a lot more people • can be more cost effective • easier recruitment • quicker turn-around • can run it over night • makes iterative development quick and simple • more diverse than university setting

  12. Diversity As of May 2009 Ross et al. [CHI 2010] • ~ 40% Bachelors, ~ 20% Graduate • ~ 50/50 gender split • 56% US, 36% India, 8% other • Maybe diverse search self-efficacy, too? • college students have high search self-efficacy (Kelly 2010)

  13. Outline • What we’re trying to do • Search self-efficacy • Searcher frustration • Search Assistance • AMT – why use it? • Initial experiments • Challenges

  14. Initial experiments – Motives • what is the spread of search self-efficacy across Turkers? • how does price / HIT affect speed and spread?

  15. HIT • HIT: search self-efficacy questionnaire • Two versions: • 100 x $0.50 • 100 x $0.05 • released at 8:30pm on two Mondays in June ...

  16. Search self-efficacy spread $0.50 / questionnaire $0.05 / questionnaire

  17. Time for all 100 HITs to be accepted

  18. Time to complete questionnaire

  19. Hourly wage Gave a bonus of $0.17 – raises median wage to: $8.55/ hour

  20. Outline • What we’re trying to do • Search self-efficacy • Searcher frustration • Search Assistance • AMT – why use it? • Initial experiments • Challenges

  21. Search self-efficacy scale challenges • modify to ask positive and negative versions of queries • allows us to check if users are paying attention • inconsistent results raise a poor-quality flag • could ask both versions of each question • very long – 26 questions • could make half positive, half negative • keeps questionnaire a manageable size

  22. Other study challenges • reducing length and complexity of stages • may be too big to overcome for this study • pricing • need a sufficient incentive for Turkers to spend so much time • quality – what’s the cost? • is Turk a reliable source for this kind of study? • what is the truthfulness of a Turker? • can we do anything to improve truthfulness? • what impact will “unreliable” data have on the results?

  23. Ethics • Is AMT exploitive? • is it just piece work? [Mieszkowski 2006] • maybe paying less than minimum wage • this could be true in non-AMT studies, too • AMT allows bonuses • can be used to increase payment based on median time to complete HIT across Turkers • ...but you don’t know exactly where that money is going • Control • no control over the Turkers’ environment • trust in AMT not necessarily the trust you’d have with an outsourcing firm

  24. Special thanks to Diane Kelly for providing us with the search self-efficacy scale and commenting on the paper.

More Related