i C r o w d : An Adaptive Crowdsourcing Framework

1. iCrowd: An Adaptive Crowdsourcing Framework Ju Fan1, Guoliang Li2, Beng Chin Ooi1, Kian-Lee Tan1, Jianhua Feng2

2. Crowdsourcing is very useful… • Humans are better than the machines at many complex jobs • Entity Resolution • Are “IBM” and “Big Blue” the same company? • Classification • Is an image pornography? • Others • Translation, Image Tagging, etc. • Crowdsourcing • Harness Human Intelligence Adaptive Crowdsourcing, SIGMOD 2015

3. But it often yields low-quality results • Crowdsourcing outsources tasks for solutions from an unknown group of people (aka workers) • Cheating: answering tasks randomly for $$$ • Attacking: providing adverse answers • Under-qualified: having low answer accuracy Adaptive Crowdsourcing, SIGMOD 2015

4. Quality Control in Crowdsourcing • A redundancy-based strategy • Assign a crowdsourcing task to multipleworkers • Derive result by aggregating worker answers • A simple solution: Majority Voting Not effective if there are many low-quality workers YES Final Result Same Product? NO NO • iPhone 6 • iPhone 6 Plus NO Task Assignment Answer Aggregation Adaptive Crowdsourcing, SIGMOD 2015

5. Considering Worker’s Quality • Offline Worker Quality Estimation • Gold-Injected: Leverage a small amount of “gold” tasks • Estimating quality as ratio of correctly completed “gold” tasks • Aggregating worker answers using probabilistic fusion • EM-Based: Exploit an Expectation-Maximization technique • Iteratively estimating worker quality and aggregating answers • Qualification test: eliminating “bad” workers • Quality-Aware OnlineTask Assignment • AskIt!, QASCA (also in this research session) Perform well in simple crowdsourcing tasks, e.g., image labeling Adaptive Crowdsourcing, SIGMOD 2015

6. Accuracy Diversity in Complicated Tasks • More complicated crowdsourcing tasks may require domain knowledge, e.g., • Evaluating quality of question-answers (QA) • Workers are only good at tasks in their familiardomains Does a user-generated answer address its question? • Question: Who first proposed Heliocentrism? • Answer: NicolausCopernicus Worker QA Domains Adaptive Crowdsourcing, SIGMOD 2015

7. Challenge & Opportunity • Challenge: Accuracy diversity makes existing solutions inadequate for producing high-quality result • Expert in some domains, spammer in others • Being good in qualification, performing bad in other tasks • Opportunity: Adaptive Crowdsourcing • Can we selectively assign tasks to workers with specific expertise to further improve the quality? Adaptive Crowdsourcing, SIGMOD 2015

8. Outline • Overview of Adaptive Crowdsourcing • The iCrowd Framework • Diverse Accuracy Estimation • A Graph-Based Approach • Online Task Assignment • An Adaptive Assignment Approach • Experimental Study • Conclusion Adaptive Crowdsourcing, SIGMOD 2015

9. iCrowd: Adaptive Crowdsourcing • Assignment is triggered by a requestfrom a worker • On-the-fly estimating worker accuracy on completed tasks • Assigning the worker a task, to which she has thehighest estimated accuracy among all online workers • Once a worker submits his answer to a task, adjusting estimation of her accuracies • Addressing the cold-start problem by leveraging some microtasks with ground truth, called qualification tasks Adaptive Crowdsourcing, SIGMOD 2015

10. Running Example • Crowdsourcing Tasks for Entity Resolution • Illustration of iCrowd Adaptive Crowdsourcing, SIGMOD 2015

11. Graph-Based Accuracy Estimation • Estimating worker’s accuracies on unassigned tasks from their similar completed tasks • Observed Accuracies : probability of answer correctness on the microtasks completed by worker • Estimated Accuracies : probability of answer correctness on the microtasks to be assigned to worker Estimation:  correctly completed will correctly complete ? Adaptive Crowdsourcing, SIGMOD 2015

12. Graph-Based Accuracy Estimation • Estimation Optimization Objectives • Objective 1: Minimize accuracy differences of similar tasks • Worker has comparable accuracies on similar microtasks • Objective 2: Minimize difference between and • Not to deviate too much from the observed accuracies • Optimization Formulation Objective 2 Objective 1 : a parameter for balancing the two objectives Adaptive Crowdsourcing, SIGMOD 2015

13. Graph-Based Accuracy Estimation • Analytical Solution of Optimization • An Iterative Approach for Computing Solution • Linearity Property: avoid online iterative computation • Offline: , given , compute • Online: Simply estimating (linear sum) Challenging Adaptive Crowdsourcing, SIGMOD 2015

14. Adaptive Microtask Assignment • Adaptively selecting top workers with higher accuracy, as estimated accuracies are continuously updated. • Top worker set : a set of top workers with the highest accuracies on microtask Accuracy • A worker may be the top worker for multiple microtasks • However, she can only complete at ONE task per time! Top Worker Sets ...... Adaptive Crowdsourcing, SIGMOD 2015

15. Optimal Microtask Assignment • Microtask Assignment • Given top worker set for each microtask , compute an optimal assignment such that each worker is assigned at most ONE microtask. • Basic Idea of OptimalAssignment • Making as many microtasks to be globally completed as possible, so as to improve accuracy estimation • Complexity: NP-Hard (reduced k-set packing) • A Greedy-Based Assignment Algorithm Adaptive Crowdsourcing, SIGMOD 2015

16. Qualification Microtask Assignment • Qualification Microtask Assignment • Preferring microtasks from which we can infer accuracies on more other microtasks  larger “influence” • Optimization: An NP-hard problem Adaptive Crowdsourcing, SIGMOD 2015

17. Outline • Overview of Adaptive Crowdsourcing • The iCrowd Framework • Diverse Accuracy Estimation • A Graph-Based Approach • Task Assignment • An Adaptive Assignment Approach • Experimental Study • Conclusion Adaptive Crowdsourcing, SIGMOD 2015

18. Experiment Setup • Datasets • YahooQA dataset: Evaluating quality of Question-Answers • ItemCompare dataset: compare two items based on a specified comparison criteria • To ensure that all approaches were compared on the same worker set, collect enough answers for each microtask from Amazon Mechanical Turk Adaptive Crowdsourcing, SIGMOD 2015

19. Experiment Setup • Baseline Approaches • We compared both task assignment and aggregation. • [RandomMV]:used a random strategy for task assignment and aggregated workers’ answers using majority voting. • [RandomEM]: used the random strategy for task assignmentand an EM algorithm to aggregate the answers. • [AvgAccPV]: estimated an average accuracy for each worker by injecting some “gold” microtasks and aggregated the answers using a probabilistic verification model. • Evaluation Metric • Accuracy = Ratio of correctly answered microtasks Adaptive Crowdsourcing, SIGMOD 2015

20. Effect of Similarity Measures • Different similarity measures did not significantly affect the performance • Cos(topic) with topic analysis achieved better performance • Similarity Measures • Jaccard • Cosine with TF-IDF weights • Cosine with “topic” weights analyzed by the LDA model Adaptive Crowdsourcing, SIGMOD 2015

21. Effect of Parameter •  all connected microtasks having the same estimated accuracies, which failed to capture accuracy diversity • is a large  estimated accuracies is very close to the observed ones, which means the estimation could not benefit from the graph-based inference Adaptive Crowdsourcing, SIGMOD 2015

22. Effect of Adaptive Assignment The proposed adaptive assignment achieved the best accuracy Adaptive Crowdsourcing, SIGMOD 2015

23. Comparison with Baselines iCrowd achieves 10% - 20% improvement on accuracy Adaptive Crowdsourcing, SIGMOD 2015

24. Efficiency of Microtask Assignment • Evaluating efficiency using simulation • iCrowd scales well in terms of number of tasks Adaptive Crowdsourcing, SIGMOD 2015

25. Conclusion • An adaptive crowdsourcing framework iCrowd • A graph-based estimation model to capture diverse accuracies of workers across different tasks • Efficient algorithms to enable instant task assignments • A system deployed on Amazon Mechanical Turk • Higher crowdsourcing quality on accuracy Adaptive Crowdsourcing, SIGMOD 2015

26. Thanks SQL Suggestion, ICDE 2011

27. Problem Formulation • Microtask: • A crowdsourcing requester publishes a set of microtasks • Worker: • A set of workers will work on the microtasks in • Worker’s Diverse Accuracy • Probability • Microtask Assignment • Each microtask is assigned to workers • Objective: maximizing overall accuracy • Limited knowledge of accuracy • Dynamic worker set Adaptive Crowdsourcing, SIGMOD 2015

28. Implementation of iCrowd on AMT Adaptive Crowdsourcing, SIGMOD 2015

29. Optimal Microtask Assignment • Discussions • Dynamical data structure for maintaining top workers • Periodical microtask assignment updates • For a worker not assigned with microtasks, actively testing by using • Qualification microtasks with ground truth • Globally completed microtask with consensus answers Adaptive Crowdsourcing, SIGMOD 2015

30. Effect of Qualification Assignment Influence-based qualification test assignment performs better Adaptive Crowdsourcing, SIGMOD 2015

31. Accuracy Diversity in Complicated Tasks • More complicated crowdsourcing tasks may require domain knowledge, e.g., • Comparing two items based on a specified criteria Which food has more calories? • Food A: Chocolate • Food B: Honey Comparison Domain Worker Adaptive Crowdsourcing, SIGMOD 2015

32. Diverse Accuracy Estimation • Observation:accuracies of a worker tend to be comparable on similarmicrotasks in same domains • Idea: Estimating worker accuracies by task similarity • Model: • graph based estimation model • Algorithm: • Estimating accuracies in a fast way Adaptive Crowdsourcing, SIGMOD 2015

33. Graph-Based Accuracy Estimation • Observed Accuracy Estimation • If is a qualification microtask: if ’s answer is equal to the ground truth; otherwise • If ’s ground truth is unknown, use voting-based method: compare ’s answer with the answers from other workers on this microtask • Discussion on Microtask Similarity • For textual microtasks, employing existing similarity metrics, e.g., Jaccard, Edit-Distance, Topic-based similarity, etc. Adaptive Crowdsourcing, SIGMOD 2015

34. Graph-Based Accuracy Estimation • Observed Accuracy Estimation • If is a qualification microtask: if ’s answer is equal to the ground truth; otherwise • If ’s ground truth is unknown, use voting-based method: compare ’s answer with the answers from other workers on this microtask • Discussion on Microtask Similarity • For textual microtasks, employing existing similarity metrics, e.g., Jaccard, Edit-Distance, Topic-based similarity, etc. Adaptive Crowdsourcing, SIGMOD 2015

35. Optimal Microtask Assignment • Formal Optimization Problem • Finding an assignment scheme s.t. • Complexity • Reduced from a NP-hard problem, k-set packing • A Greedy-Based Assignment Algorithm Adaptive Crowdsourcing, SIGMOD 2015