1 / 29

Crowd Centrality

Crowd Centrality. David Karger Sewoong Oh Devavrat Shah MIT and UIUC. Crowd Sourcing. Crowd Sourcing. $30 million to land on moon. $0.05 for Image Labeling Data Entry Transcription. Micro-task Crowdsourcing. Micro-task Crowdsourcing. Left. Left.

helki
Download Presentation

Crowd Centrality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Crowd Centrality David KargerSewoong Oh Devavrat Shah MIT and UIUC

  2. Crowd Sourcing

  3. Crowd Sourcing $30 million to land on moon $0.05 for Image Labeling Data Entry Transcription

  4. Micro-task Crowdsourcing

  5. Micro-task Crowdsourcing Left Left Which door is the women’s restroom? Right

  6. Micro-task Crowdsourcing Find cancerous tumor cells Reliability 90% 65% Undergrad Intern: Mturk (single label): 200 image/hr, cost: $15/hr 4000 image/hr, cost: $15/hr 90% 500 image/hr, cost: $15/hr Mturk (mult. labels):

  7. The Problem • Goal: • Reliable estimate the tasks with minimal cost • Operational questions: • Task assignment • Inferring the “answers”

  8. Task Assignment Tasks • Random ( , )-regular bipartite graphs Batches • Locally Tree-like • Sharp analysis • Good expander • High Signal to Noise Ratio

  9. Modeling The Crowd + - + - + • Binary tasks: • Worker reliability: • Necessary assumption: we know Aij

  10. Inference Problem ti • Majority: • Oracle: + - + p2 p1 p3 p4 p5

  11. Inference Problem • Majority: • Oracle: • Our Approach: p2 p1 p3 p4 p5

  12. Preview of Results Distribution of {pj}: observed to be Beta distribution by Holmes ‘10 + Ryker et al ‘10 EM algorithm : Dawid, Skene ‘79 + Sheng, Provost, Ipeirotis ‘10

  13. Preview of Results

  14. Iterative Inference • Iteratively learn • Message-passing • O(# edges) operations • Approximate MAP p2 p1 p3 p4 p5

  15. Experiments: Amazon MTurk • Learning similarities • Recommendations • Searching, …

  16. Experiments: Amazon MTurk • Learning similarities • Recommendations • Searching, …

  17. Experiments: Amazon MTurk

  18. Task Assignment: Why Random Graph

  19. Key Metric: Quality of Crowd Crowd Quality Parameter • Theorem (Karger-Oh-Shah). • Let n tasks assigned to n workers as per • an (l,l) random regular graph • Let ql > √2 • Then, for all n large enough (i.e. n =Ω(lO(log(1/q))elq))) after O(log (1/q)) iterations of the algorithm p2 p1 p3 p4 p5 • If pj = 1 for all j • q = 1 • If pj= 0.5 for all j • q =0 • q different from • μ2 = (E[2p-1])2 • q≤μ≤√q

  20. How Good Is This ? • To achieve target Perror ≤ε, we need • Per task budget l = Θ(1/q log (1/ε)) • And this is minimax optimal • Under majority voting (with any graph choice) • Per task budget required is l = Ω(1/q2 log (1/ε)) no significant gain by knowing side-information (golden question, reputation, …!)

  21. Adaptive Task Assignment: Does it Help ? • Theorem (Karger-Oh-Shah). • Given any adaptive algorithm, • let Δbe the average number of workers required per task • to achieve desired Perror ≤ε • Then there exists {pj} with quality q so that gain through adaptivity is limited

  22. Which Crowd To Employ

  23. Beyond Binary Tasks • Tasks: • Workers: • Assume pj ≥ 0.5 for all j • Let q be quality of {pj} • Results for binary task extend to this setting • Per task, number of workers required scale as • O(1/q log (1/ε) + 1/q log K) • To achieve Perror ≤ ε

  24. Beyond Binary Tasks • Converting to K-1 binary problems • each with quality ≥ q • For each x, 1 < x ≤ K: • Aij(x) = +1 if Aij≥ x, and -1 otherwise • ti(x) = +1 if ti≥ x, and -1 otherwise • Then • Corresponding quality q(x) ≥ q • Using result for binary problem, we have • Perror(x) ≤ exp(-lq/16) • Therefore • Perror ≤ Perror(2) + … + Perror(K) ≤ K exp(-lq/16)

  25. Why Algorithm Works? • MAP estimation • Prior on probability {pj} • Let f(p) be density over [0,1] • Answers A=[Aij] • Then, • Belief propagation (max-product) algorithm for MAP • With Haldane prior: pjis0 or 1 with equal probability • Iteration k+1: for all task-worker pairs (i,j) Xi/Yjrepresent log likelihood ratio for ti/pj= +1 vs-1 • This is exactly the same as our algorithm! • And our random task assignment graph is tree-like • That is, our algorithm is effectively MAP for Haldane prior

  26. Why Algorithm Works? • A minor variation of this algorithm • Tinext = Tijnext= ΣWij’ Aij’ = ΣWj’ Aij’ • Wjnext= Wijnext= ΣTi’jAi’j = ΣTi’ Ai’j • Then, • Tnext= AAT T • (subject to this modification) our algorithm is computing • Left signular vector of A (corresponding to largest s.v.) • So why compute rank-1 approximation of A ?

  27. Why Algorithm Works? • Random graph + probabilistic model • E[Aij] = (tipj- (1-pj)ti) l/n =ti(2pj-1)l/n • E[A] = t (2p-1)T l/n • That is, • E[A] is rank-1 matrix • And, t is the left singular vector of E[A] • If A ≈ E[A] • Then computing left singular vector of A makes sense • Building upon Friedman-Kahn-Szemeredi ‘89 • Singular vector of A provides reasonable approximation • Perror = O(1/lq) • Ghosh, Kale, Mcafee’12 • For sharper result we use belief propagation

  28. Concluding Remarks • Budget optimal micro-task crowd sourcing via • Random regular task allocation graph • Belief propagation • Key messages • All that matters is quality of crowd • Worker reputation is not useful for non-adaptive tasks • Adaptation does not help due to fleeting nature of workers • Reputation + worker id needed for adaptation to be effective • Inference algorithm can be useful for assigning reputation • Model of binary task is equivalent to K-ary tasks

  29. On That Note…

More Related