Micro Tasks in Human Computation for Computer Graphics

1. Micro Tasks in Human Computationfor Computer Graphics Ariel Shamir With: Yotam Gingold, Daniel Cohen-Or (TAU)

2. Historical Note: Computers? • Before iPhones, PCs and laptops, the term "computer" referred to the people who did scientific calculations by hand. • These workers (mostly women) were neither calculating geniuses nor idiot savants but knowledgeable people who, in other circumstances, might have become scientists in their own right! • Many highly educated women of that generation studied to become a human computer because nothing else would offer them a place in the scientific world.

3. When Computers Were Human • The sad but lyrical story of workers who gladly did the hard labor of research calculation in the hope that they might be part of the scientific community. • In the end, they were rewarded by a new electronic machine that took the place and the name of those who were, once, the computers.

4. (Electronic) Computers • Deterministic • Fast • Efficient for calculations

5. (Electronic) Computers (Most) Humans • Deterministic • Fast • Efficient for calculations • Inconsistent • Slow • Still better at some tasks (many? which?)

6. Humans are Better at… (Disclaimer: my own take on this) • Self consciousness (singularity is near?) • Emotions • Learning and adapting • Selecting the main from the unimportant • Perception

7. Turing Test • A test of a machine's ability to exhibit intelligent behavior. • A human judge engages in a natural language conversation with one human and one machine, each of which tries to appear human. All participants are separated from one another. • If the judge cannot reliably tell the machine from the human, the machine is said to have passed the test. ? ?

8. CAPTCHA • Sometimes described as a Reverse Turing Test. • Completely Automated Public Turing test to tell Computers and Humans Apart (Luis von Ahn, Manuel Blum, Nicholas J. Hopper, and John Langford, CMU 2000, based on ideas from MoniNaor) • A type of challenge-response test used in computing as an attempt to ensure that the response is generated by a person. • The process usually involves one computer asking a user to complete a simple test which the computer is able to generate and grade. ? ?

9. Digitizing Books One Word at a Time • Each new word that cannot be read correctly by OCR is given to a user in conjunction with another word for which the answer is already known. The user is then asked to read both words. If they solve the one for which the answer is known, the system assumes their answer is correct for the new one. The system then gives the new image to a number of other people to determine, with higher confidence, whether the original answer was correct.

10. Human Computation • Luis Von Ahn, PhD thesis, CMU 2005 • A computational process that performs its function by outsourcing certain steps to humans. • This approach uses differences in abilities and alternative costs between humans and computer agents to achieve symbiotic human-computer interaction.

11. Issues • Motivation: payment, fun (any other?) • Games with a Purpose (GWAP) • Mechanical Turk • Cost & Efficiency • Quality Control: • Duplication • Sentinel Operations • Self Referring • Granularity

12. Example 1: • Motivation: Fun! • Quality control: only matching words • Main use: tagging images (good for google!) • Now aka http://www.gwap.com/gwap/gamesPreview/espgame/

13. Example 2: Instructions: • Start by pressing the left mouse button at some point along the boundary of the object.  • Continue clicking along the boundary of the object to create a polygon.  • Once you have finished clicking along the boundary of the object, either click on the first point or press the right mouse button to complete the polygon.  • A window will now appear asking for the object's name. Enter the object's name and click the "Done" button. http://labelme.csail.mit.edu/

14. Example 3: An add-in to Microsoft Word that uses crowd contributions to perform interactive document shortening, proofreading, and human-language macros. Underlying Soylent is a new programming design pattern called Find-Fix-Verify that splits tasks into a series of generation and review stages to control costs and increase quality.

15. Granularity? • A word • A phrase • A paragraph • A whole paper • A whole book

16. Humans using Computers • There is a tradeoff between how much work the human does and how much work the computer does

17. Range of Solutions • You can think of it as a slider in terms of How much human and how much computer is involved More Computer “cycles” More Human “cycles” Fully Automatic (no human) Human Computation Interactive Application Let the Human do it

18. Type of Human Cycles • But you can also think of it as the range of type of activity the human does: High level Complex Human Computation Low level Fine grain Human Computation Interactive Application Our Model Label-Me ESP

19. Our Model • Very simple and very specific cognitive task. • In a sense, we want to distill the human computation to the bare basic and use it for what the humans are most effective at (and computers are not!): • Must be a cognitive task • Ask a very simple question • Ask a very specific question • This model leads to highly parallel design

20. The Question We Ask • What is the minimum amount of information a human can give the computer in order to solve the task? • Rephrasing the algorithm in terms of the smallest piece of information that without it the problem could not be solved.

21. Micro Tasks • When dealing with graphics problems the key characteristic that still provides an advantage to humans is perception, and very often visual perception. • HC tasks based on visual perceptual queries! • No training or skill needed • No dependency (between tasks) • Highly parallel

22. Algorithm Design Pattern

23. Example: Depth • Depth (distance from the camera) is an important cue that can assist image manipulations (insertion and removal of objects, de-hazing, retargeting etc.) • Today: depth cameras

24. Calculate Depth of a Given Image? • Automatic methods: • Depth increases in the up direction • Color similarity implies depth similarity • Not always correct

25. Calculate Depth of a Given Image? • Automatic methods: • Depth increases in the up direction • Color similarity implies depth similarity • Not always correct • Some images are very challenging (art)

26. Micro Task? • Ask “what is the depth of the pixel?” • Too fine, can be ambiguous • Ask “what is the depth of an object?” • Too complex, may take time • Ask “what is the depth of a patch in the image?” • Humans are not good at assessing absolute depth

27. Relative Ordering • Ask between two patches: which is closer? • Still ambiguous – A closer than B but:

28. Our Micro Task

29. Guidelines for Choosing Tasks • Task must be simple (instantaneous) • Task must be specific (accurate) • Task must be cognitive (computers can’t solve it today)

30. Combining • The relative depth ordering provides offsets of -1, 0, or 1 between adjacent regions in the image • To reconstruct a continuous depth map we solve a Laplace equation with derivative constraints of -1, 0, or 1 across region boundaries.

31. Algorithm Partition Quality Setup Dispatch Verify Combine

34. Finding Normals

36. Computational Model • Just like there is cloud computing, you can imagine a “HP cloud” where you submit micro-tasks and receive answers (by paying some fee): HP Cloud Algorithm Submit Queries Receive Answers

37. Some Statistics

38. Timing

39. Computing Cloud • Behind this cloud someone can develop an algorithm that can imitate humans (and gain money) – and it would be fine! Computing Cloud Algorithm Submit Queries Receive Answers

40. Future… • Reversing the roles: instead of PDA (personal digital assistance) robots will have their own PHA. • Thank You! • PHA = personal human assistance that will assist in solving minimal HC tasks.