SCG Court: A Crowdsourcing Platform for Innovation

1. SCG Court: A Crowdsourcing Platform for Innovation Supported by Novartis • Karl Lieberherr • Northeastern University • College of Computer and Information Science • Boston, MA • joint work with Ahmed Abdelmeged

2. SOLVE ORGANIZATIONAL PROBLEMS HOW TO COMBINE THE WORK OF HUNDREDS OF SCHOLARS? HOW TO AVOID ECONOMIC NON-SENSE HOW TO FOCUS SCHOLARS Crowdsourcing

3. Organizational Problem Solved • How to organize a loosely coupled collaboration among several scholars to agree on claims that can be refuted or defended constructively using a dialog. • fair recognition of scholars • strong scholars cannot be ignored • output: answer: “is claim refuted” plus the dialog • When game is over: interested in • know-how! • list of claims that scholars agree with. defend(Alice,Bob,c) = ! refute(Alice,Bob,c) Crowdsourcing

4. happy = no scholar is ignored, rich, immersive experience. Organizational Problem Solved • How to design a happy scientific community that creates the science that society needs. • Classical game solution: Egoistic scholars produce social welfare: knowledge base and know-how how to defend it. • Control of scientific community • SCG rules • Specific domain Crowdsourcing

5. What is a loose collaboration? • Scholars can work independently on an aspect of the same problem. • Problem = decide which claims in playground to oppose or agree with. • How is know-how combined? Using a protocol. • Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. • Bug reports that need to be addressed and corrections. Playground = Instantiation of Platform Crowdsourcing

6. Controlled Communication instead of Isolation s: scholar s1 s2 s1 s2 admin admin s3 s4 s3 s4 more learning and collaboration team evaluation undesirable Crowdsourcing

7. Claims • Protocol. Defines scientific discourse. • Scholars make a prediction about their performance in protocol. • Predicate that decides whether refutation is successful. Refutation protocol collects data for predicate. • As a starter: Think of a claim as a mathematical statement: EA or AE. • all planar graphs have a 4 coloring. Crowdsourcing

8. Benefits for Playground Designers Return On Investment for playground designers: a small investment in defining a playground (Domain=(Instance,Solution,valid,quality), Claim=(Protocol, etc.)) produces an interactive environment to assimilate and create domain knowledge. Crowdsourcing

9. Benefits for Playground Designers Organize the thought processes of hundreds of scholars to creatively focus on one problem Playground Design Financial Incentive Crowdsourcing

10. Benefits for Scholars • Return on Investment for scholars and avatar designers: The SCG rules need to be learned only once because they are the same across playgrounds. A small investment in learning the SCG rules and a domain leads to numerous learning and teaching and innovation opportunities. The more a scholar teaches, the higher the scholar’s reputation. Crowdsourcing

11. Benefits for Scholars: Immersion Experience scholar s s I challenge you (propose) show me (solve) me you are wrong (oppose, teaching and learning) tell me (provide) s s Innovation: Scholars are free to invent; game rules don’t limit creativity; social engineering: answers to “why did I lose?” may lead to better solutions. Crowdsourcing

12. Global Warming • Alice’ Claim: The earth is warming significantly. • Refutation protocol: Bob tries to refute. • Alice must provide a data set DS satisfying a property defined precisely by the refutation protocol. • Bob applies one of the allowed analysis methods M defined precisely by the refutation protocol. • Bob wins iff M(DS) holds. Crowdsourcing

13. Independent Set • Protocol / claim: At Least As Good • Bob provides undirected graph G. • Bob computes independent set sB for G (secret). • Alice computes independent set sA for G. • Alice wins, if size(sA) >= size(sB). Crowdsourcing

14. Overview • Organizational problem that SCG solves • What is SCG in detail? • Crowdsourcing • Formal Properties of SCG • Applications • Disadvantages • Conclusions Crowdsourcing

15. Big Picture • Weaker form of logic. • Approximate truth. • Don’t focus on proofs but on refutations. Crowdsourcing

16. Logic with Soundness claims sentences good bad not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum Crowdsourcing

17. Scientific Community Game Logic with Community Principle claims sentences good bad disagreed by two scholars agreed by two scholars agree c(informally): both successfully defend c and both successfully refute negation !c. there exists two-party certificate to expose misclassification Crowdsourcing

18. Comparison Logic and SCG Logic Scientific Community Game sentences = claims good bad evidence for goodness defense, checkable uncertainty of defense evidence for badness refutation, checkable uncertainty of refutation Personified sentences • sentences • true • false • proof for being true • proof system, checkable • guaranteed defense • proof for being false • proof system, checkable • guaranteed refutation • Universal sentences Crowdsourcing

19. Example: Highest Safe Rung • You are doing stress-testing on various models of glass jars to determine the height from which they can be dropped and still not break. The setup for this experiment, on a particular type of jar, is as follows. Crowdsourcing

20. Highest Safe Rung admin Bob Alice You have a ladder with n rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung. You have a fixed ``budget'' of k > 0 jars. Only two identical bottles to determine highest safe rung Crowdsourcing

21. Highest Safe Rung Bob Alice HSR(9,2)≤4/9 I doubt it: refutation attempt! Alice constructs decision tree T of depth 4 and gives it to Bob. He checks whether T is valid. Bob wins if he finds a flaw. Only two identical bottles to determine highest safe rung Crowdsourcing

22. x Highest Safe Rung Decision Tree HSR(10,2)=5/10 no 3 yes y z 1 6 u highest safe rung 0 2 4 9 1 2 3 5 7 9 4 5 Other playgrounds might have only two claims: C and !C. 8 6 7 8 Crowdsourcing

23. HSR(n,k)=q/n abbreviate as HSR(n,k)=k n: rungs k: to break q: questions Scientific Community Game Logic with Community Principle claims sentences good bad HSR(8,3)=3 HSR(17,4)=4 HSR(11,2)=4 HSR(27,3)=5 HSR(5,1)=4 HSR(15,3)=5 HSR(57,4)=6 disagreed by two scholars agreed by two scholars there exists two-party certificate to expose misclassification Crowdsourcing

24. Community Principle for SCG • Every faulty decision has a certificate to assign blame to the faulty decision maker. • Certificate contains information from both parties. • A certificate is a sequence of moves leading to a loss for the scholar making the faulty decision. • The certificate can be checked efficiently assuming all basic game operations (valid, belongsTo, quality) take constant time. Crowdsourcing

25. Example • HSR(15,3)=5 • This claim was misclassified as a good claim because both Alice and Bob could only find a decision tree of depth 5. • Nina, a newcomer to the HSR scientific community, could find a decision tree of depth 4. • Exposing the faulty decision of Alice and Bob. Crowdsourcing

26. Big Picture • Replace soundness with community principle. • Participants have to work hard to approximate soundness; if they don’t achieve soundness, they risk to be caught and risk to lose reputation. • as in a real scientific community: mistakes are made, even in mathematical journals. Crowdsourcing

27. What is SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge Crowdsourcing

28. A Virtual WorldAvatar’s View Avatar does complex work Claims, Instances, Solutions Opponents’ communication, Feedback Administrator does simple checking (usually efficient) Results • Agreed Claims: statements about algorithms • = Social welfare • Algorithms in Avatar Crowdsourcing

29. Life of an avatar: (propose+ (oppose | agree)+ provide* solve*)* Avatars propose and (oppose|agree) proposed claims egoistic Alice egoistic Bob CB1 opposes (1) CA1 CB2 provides instance (2) CA2 solves instance LOSES WINS! not as well as she expected based on CA2 (3) CA3 reputation 10 reputation 1000 transfer 200 CA4 Crowdsourcing

30. What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? Crowdsourcing

31. blamed decisions: propose(Alice,c) refute(A,B,c) strengthen(Alice,Bob,c,cs) agree(A,B,c) Essence of Game Rules • actors: • proposer=verifier (1. arg to propose, oppose, refute, usually Alice), • opposer=falsifier (2. arg to propose, oppose, refute, usually Bob) • LifeOfClaim(c) = propose(Alice,c) followed by (oppose(Alice,Bob,c)|agree(Alice,Bob,c)). • oppose(Alice,Bob,c) = (refute(Alice,Bob,c)|strengthen(Alice,Bob,c,cs)), where stronger(c,cs). • strengthen(Alice,Bob,c,cs) = !refute(Bob,Alice,cs). • agree(Alice,Bob,c) = !refute(Alice,Bob,c) and !refute(Bob,Alice,c) and refute(Alice,Bob,!c) and refute(Bob,Alice,!c) Crowdsourcing

32. Winning/Losing • propose(Alice,c), refutationTry(Alice,Bob,c) • If Alice first violates a game rule, Bob is the winner. • If Bob first violates a game rule, Alice is the winner. • If none violate a game rule: the claim predicate c.p(Alice,Bob,in,out) decides. Crowdsourcing

33. Game Rules for Playground • legal(in) • legal(out) • valid(in,out) • belongsTo(in, instanceSet) • each move must be within time-limit Crowdsourcing

34. Protocol Language ProtocolSpec = <steps> List(Step). Step = <action> Action "from" <role> Role. interface Role = Alice | Bob. Alice = "Alice". Bob = "Bob". interface Action = ProvideAction | SolveAction. ProvideAction = "instance". // solve the instance provided in // step # stepNo. // stepNo is 0-based. SolveAction = "solution" "of" <stepNo> int. instance from Bob // r solution of 0 from Bob // sB for r solution of 0 from Alice // sA for r Crowdsourcing

35. How to achieve loosely coupled collaboration? • Information exchange is based on values. Knowledge how to produce values is secret. • Assign blame correctly to Alice or Bob based on outcome of refutation protocol. • Every claim has a negation (using the idea of Hintikka’s dual game). • Negation of HSR(n,k)=q: HSR(n,k)<q. Crowdsourcing

36. Dual Game / Negation • Each game G has a dual game which is the same as G except that the players Alice and Bob are transposed in both the rules for playing and the rules for winning. The game G(¬φ) is the dual of G(φ). Crowdsourcing

37. How is collaboration working? • Scholars make claim about their performance in a given context. • Scholars make claim about the performance of their avatar in a given context. • Opponent finds input in context that contradicts claim. Claim is refuted. Crowdsourcing

38. Playground Design • Define several languages • Instance • Solution • Claim • InstanceSet • Define protocol or reuse existing protocol. • Implement interfaces for corresponding classes. Crowdsourcing

39. Who are the scholars? • Students in a class room • High school • University • Members of the Gig Economy • Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. • Potential employees • Anyone with web access; Intelligent crowd. Crowdsourcing

40. How to engage scholars?Opposition • Central to opposition is refutation. • Claim defined by protocol. • Simplest protocol: • Alice provides Input in. • Bob computes Output out: valid(in,out) • Alice defends if quality(in,out)<q. • Bob refutes if quality(in,out)>=q. • Claims: C(q), q in [0,1]. instance from Alice // in solution of 0 from Bob // out for in Crowdsourcing

41. Overview • Organizational problem solved by SCG • What is SCG in detail? • Crowdsourcing • Formal Properties of SCG • Applications • Disadvantages • Conclusions Crowdsourcing

42. Crowdsourcing • Active area: Recent Communication of the ACM article. • Wikipedia, FoldIt, TopCoder, … • We want a family of crowdsourcing systems with provable properties. Crowdsourcing

43. Crowdsourcing Platform • Crowdsourcing • is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call. • enlists a crowd of humans to help solve a problem defined by the system owners. • A crowdsourcing platform is a generic tool that makes it easy to develop a crowdsourcing system. Crowdsourcing

44. Crowdsourcing Platform • The job, target problem is • to solve instances of a problem and make claims about the solution process. • to build knowledge base of claims and techniques to defend the claims Crowdsourcing

45. Requirements for Crowdsourcing Platform • Find a good way to combine user contributions to solve the target problem. • Find a good way to evaluate users and their contributions. • Find a good way to recruit and retain users. Crowdsourcing

46. SCG Court • Web application • Software developers register with SCG Court and choose playgrounds they want to compete in. • They register their avatars in the appropriate playgrounds in time for the next tournament. • Avatars get improved between tournaments based on ranking achieved and game history. Crowdsourcing

47. Combine user contributions • Users build on each others work: strengthening and checking. • Users check each others claims for correct judgment. • Claims are defended and refuted. • Users trade reputation for information. • Example: HSR(15,3)=5 Crowdsourcing

48. Learning cycle • Alice wins reputation with claim c because Bob made a wrong decision • Alice gives information about artifact related to c. Alice teaches Bob. • Bob integrates information into his know-how. Bob learns from Alice. • Bob hopefully has learned enough and will no longer make a wrong decision about c. Crowdsourcing

49. Voting with Justification • I vote • for this claim (agree) because I can defend it and refute its negation. • against this claim because I can oppose it (refute or strengthen). Crowdsourcing

50. Evaluate users and their contributions • Calculate reputation • confidence by the proposer that a claim is good (gc) • confidence by the opposer (refute or strengthen) that the claim is bad (bc) • The scholars are encouraged to set their confidences truthfully. Otherwise they don't gain enough reputation or they lose too much reputation. Crowdsourcing