150 likes | 302 Views
Using Mechanical Turk for linguistic research. Fast, cheap, easy, and reliable. Tyler Schnoebelen, Stanford University Victor Kuperman, McMasters University. The Cloze task. Used by psycholinguists to understand contextual constraints Stimuli 1: Margie ___
E N D
Using Mechanical Turk for linguistic research Fast, cheap, easy, and reliable Tyler Schnoebelen, Stanford University Victor Kuperman, McMasters University
The Cloze task • Used by psycholinguists to understandcontextualconstraints • Stimuli 1: Margie ___ • Stimuli 2: Margie moved ___ • Stimuli 3: Margie moved into ___ • Summary: • Turk and students have a highcorrelation (rho=0.823) and performsimilarly (48 stimuli sentences, all with 10+ words)
Semantic similarity • Phrasal verbs are sometimes transparent in their meaning: • You lift up a piece of paper and you “lift” and the paper goes “up” • But sometimes opaque: • If you bring up children, the children do not really get brought anywhere (or go up except metaphorically) • We collected judgments on 96 phrasal verbs • Without context, rate them for similarity 1-7 • cool cool down _______________ • Or with context: • The fan will cool down the engine when it starts to heat up. • cool cool down _______________
References • Crowdsourcing and language studies: the new generation of linguistic data. Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, CA • Munro, Robert, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen and Harry Tily. 2010. • We’ll be presenting at the LSA in Pittsburgh this January, too • Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks, Proceedings of the Conference on Empirical Methods in Natural Language Processing, October 25-27, 2008, Honolulu, Hawaii • RionSnow , Brendan O'Connor , Daniel Jurafsky , Andrew Y. Ng, • "Preventing Satisficing in Online Surveys: A 'Kapcha' to Ensure Higher Quality Data" • Adam Kapelner(University of Pennsylvania) • Dana Chandler