The future of crowd work Aniket Kittur et al., (2013). The future of crowd work , CSCW’13

1. The future of crowd workAniketKittur et al., (2013). The future of crowd work, CSCW’13 MejdlSafran Fall 2013 SIUC

2. More complex, creative and highly valued work “a platform is needed for managing pools of tasks and workers. Complex tasks must be decomposed into smaller subtasks, each designed with particular needs and characteristics which must be assigned to appropriate groups of workers who themselves must be properly motivated, selected (e.g., through reputation), and organized (e.g., through hierarchy). Tasks may be structured through multi-stage workflows in which workers may collaborate either synchronously or asynchronously. As part of this, AI may guide (and be guided by) crow workers. Finally, quality assurance is needed to ensure each worker’s output is of high quality and fits together.”

3. Research challenges in 12 major areas • The future of crowd work processes • The future of crowd computation • The future of crowd workers

4. Crowd Work: Workflow Motivation/Goals • Complex crowd work cannot accomplished by simple parallel approaches (e.g., aggregating multiple independent judgments) • Workflows are needed that facilitate decomposing tasks into subtasks, managing the dependencies between subtasks, and assembling the results.

5. Crowd Work: Workflow Research Proposal • Improving existing workflows toward more general tasks and complex problems that have no clearly defined solution (creativity, civic planning, etc.) • Comparing to simple workflow, crowd workflow could involve a much larger scale of operation, an a much more heterogeneous set of actors • How to validate? • We must experiment on a large space of parameters, incentives and decompositions. • Costs of doing so can be reduced through models of worker behavior, or by reusing proven design patterns. .

6. Crowd Work: Task Assignment Motivation/Goals • Sharing limited resources (workers) requires coordination. • While workers are continuously employed with tasks that match their interests, requesters will see their tasks completed quickly. • ESP game and Galaxy Zoo use FCFS model • oDesk and AMT use market model

7. Crowd Work: Task Assignment Research Proposal • Data partitioning, OS scheduling and failover! • Or new theoretical model. • “Several workers in our survey complained that they spent too much energy finding appropriate tasks” (Task Recommendation). • Research question: should tasks be pulled by workers or pushed by platforms?

8. Crowd Work: Quality Control Motivation/Goals • Why? High throughput, low transaction costs ands complex/subjective tasks. • Cheating or game the system. • Workers with low expertise and requesters with unclear instructions.

9. Crowd Work: Quality Control Research Proposal • Up-front task design: peer-review, agreement filters (writing an essay!!), optimize instructions, etc. • Post-hoc result analysis: gold standard data (prevent biases from dominating the results) and workers’ votes (costs more). • Better approach: use ML to predict the quality of a worker’s output from their behavior. • Open-ended work and highly skilled tasks?? • Is it possible to robustly measure worker’s skills at tasks such as audio engineering and poetry? Should we rely on peer evaluation?

10. Crowd Work: Real-time Crowd Work Motivation/Goals • We need to create flash crowds: groups of individuals who arrive moments after a request and can work synchronously. • All on-demand CS are limited by the crowd latency.

11. Crowd Work: Real-time Crowd Work Research Proposal • Fast recruitment is a major research area in real-time CS. • Time-limited tasks (e.g., search for a missing person and timed competitions) • E-mail a set of workers the night before the study and announced a time for the experiment. • Paying workers a small wage to stay on call. • They need to be modeled using queuing theory. • Scaling up to increased demand for real-time workers • Making workers efficient enough to collectively generate results ahead of time deadlines

12. Crowd Computation: Crowds Guiding AIs Motivation/Goals • In human computation, people act as computational components and perform the work that AI systems lack the skills to complete. • By tapping into crowd intelligence, computational systems can support a much broader set of tasks • Crowd workers may end up training machines to replace them!! • Gathered data is used to train algorithms (e.g., Google image search engine, birds count project)

13. Crowd Computation: Crowds Guiding AIs Research Proposal • Integrating crowds more deeply into algorithms. • Design ML algorithms that more deeply understand the human nature of these labels.

14. Crowd Computation: AIs Guiding Crowds Motivation/Goals • Humans can introduce errors, and errors are amplified as they propagate through the crowd. • Integrate crowds inside of software and use the software to help guide crowd work. • E.g., ML model can determine which work products may still be improved and then assign workers most likely to make such improvements.

15. Crowd Computation: AIs Guiding Crowds Research Proposal • AIs can point out what others have done in similar contexts. • We need to move from a setting where simple AIs completely determine a workflow to a richer, mixed setting where crowds and AIs jointly teach each other, and jointly control the work process.

16. Crowd Computation: Crowdsourcing Platforms Motivation/Goals • Limiting ourselves to exiting platforms greatly restricts the scope and nature of change we can enact.

17. Crowd Computation: Crowdsourcing Platforms Research Proposal • Different platforms for different aims. • MobileWorks (mClerk): through SMS, more populations. • Vending machine with purpose (exam grading) targeting experts. • Labor exchanged  policy issues • How can platforms disclose enough info to be trusted as a source of worker quality while also maintaining privacy? • Security concerns both attacks on platforms or use of platforms for launching attacks

18. Crowd Workers: Reputation Motivation/Goals • “Graduated from UC Berkeley and working Google” • Traditional employers judge a prospective employee's education and work history through interviews, transcripts, and references. • “I have an accepting rate of 4 out of 5 on AMT”

19. Crowd Workers: Reputation Research Proposal • We want better reputation rankings for workers to be established with the platforms • Reputation systems must be robust to cheating and gaming while supporting low-transaction cost hiring. • One way: a web of trust! (workers and requesters validate each other. • What if malicious build their own web of trust!!! (gaming the system) • Technical tools for sharing information about workers  requester abuses!! (privacy)

20. Crowd Workers: Motivation & Rewards Motivation/Goals • Crowd workers are a diverse and multifaceted population with a range of motives and experiences • Worker said “Mturk should encourage and reward requesters that provide clear instructions, quick payment, and higher pay”

21. Crowd Workers: Motivation & Rewards Research Proposal • Past research: • Requesters should clearly understand and communicate desired behaviors • Understand and worker motivations and incentives with these desired behaviors • Design the requests and incentive structures in order to achieve both effective task completion and worker satisfaction • Beyond financial incentives • Build systems that support workers’ diverse motivations.