Soylent

1. Soylent A Word Processor with a Crowd Inside Kyong Hwa Bae

2. Outline • What is Soylent? • Features of Soylent • Techniques for making Soylent • Evaluation • Discussion

3. What’s Soylent? • A word processing interface that utilizes crowd contributions • To aid writing tasks – error prevention, paragraph shortening, and automation of tasks like citation searches and tense changes

4. Soylent

5. Features • Shortn : Text Shortening • Crowdproof : Crowdsourced Proofreading • The Human Macro : Natural Language Crowd Scripting

6. Challenges in Programming with Crowd Workers • High Variance of Effort • Lazy Turker • Eager Turker • Turkers Introduce Errors 30% error the raw results from Turkers!

7. The theme of loneliness features throughout many scenes in Of Mice and Men and is often the dominant theme of sections during this story. This theme occurs during many circumstances but is not present from start to finish. In my mind for a theme to be pervasive is must be present during every element of the story. There are many themes that are present most of the way through such as sacrifice, friendship and comradeship. But in my opinion there is only one theme that is present from beginning to end, this theme is pursuit of dreams The theme of loneliness features throughout many scenes in Of Mice and Men and is often the principal, significant, primary, preeminent, prevailing, foremost, essential, crucial, vital, criticaltheme of sections during this story

8. Techniques for Programming Crowds • Find-Fix-Verify Pattern • Find : to identify patches that need to be edited • Fix : to revise an identified patch • Verify : to control the quality of revisions • Find-Fix-Verify in Soylent • Shortn and Crowdproof use the pattern.

9. Find-Fix-Verity Word Mechanical Turk Find “Identify” Fix “Edit” Verify “Choose”

10. Shortn Evaluation • Goal : to see how much Shortn could shorten text, as well as its cost and time • Results : • 78% ~ 90% of the original document length (50% after 3 iterations) • 18.5 minutes of wait time + 2 minutes of work time • $1.41 per paragraph

11. Shortn Evaluation • Issues : • Shortn occasionally introduces errors. (ex) removing the academic phrase “In this paper we argue that” => stylistically incorrect, capitalization and punctuation problems • Parallelism makes errors. (ex) “the low order bits of the key and next..(the key fragment)” => unmatched cuts

12. Crowdproof Evaluation • Results : • 88% of error fixing ( 67% error caught from Soylent, 30% from Microsoft Word ) • 18 minutes of wait time + 2.8 minutes of work time • $3.40 per paragraph • Issues : • Lazy Turkers don’t fix all errors. • A lack of domain knowledge (ex) GUIs => GUI

13. Human Macro Evaluation • Goals : • Intension – Did Turkers understand a task? • Accuracy – Was the result flawless? • Results : • 88% of intension success ( typical errors occur when ask two requirements ) • 70.8% of accuracy success ( failed on some detail or too much information from Eager Beavers )

14. Discussion • Wait time • Cost • Privacy • Legal ownership • Domain knowledge

15. END Kyong Hwa Bae