1 / 96

SCG Court: A Crowdsourcing Platform for Innovation

SCG Court: A Crowdsourcing Platform for Innovation. Supported by Novartis. Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA joint work with Ahmed Abdelmeged. SOLVE ORGANIZATIONAL PROBLEM HOW TO COMBINE THE WORK OF HUNDREDS

questa
Download Presentation

SCG Court: A Crowdsourcing Platform for Innovation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SCG Court: A Crowdsourcing Platform for Innovation Supported by Novartis • Karl Lieberherr • Northeastern University • College of Computer and Information Science • Boston, MA • joint work with Ahmed Abdelmeged

  2. SOLVE ORGANIZATIONAL PROBLEM HOW TO COMBINE THE WORK OF HUNDREDS OF SCHOLARS? Crowdsourcing

  3. Organizational Problem Solved • How to organize a loosely coupled collaboration among several scholars to agree on claims that can be refuted or defended constructively using a dialog. • fair recognition of scholars • strong scholars cannot be ignored • output: answer: “is claim refuted” plus the dialog • When game is over: interested in • know-how! • list of claims that scholars agree with. defend(Alice,Bob,c) = ! refute(Alice,Bob,c) Crowdsourcing

  4. happy = no scholar is ignored. Organizational Problem Solved • How to design a happy scientific community that creates the science that society needs. • Classical game solution: Egoistic scholars produce social welfare: knowledge base and know-how how to defend it. • Control of scientific community • SCG rules • Specific domain, claim definition to narrow scope. Crowdsourcing

  5. What is a loose collaboration? • Scholars can work independently on an aspect of the same problem. • Problem = decide which claims in playground to oppose or agree with. • How is know-how combined? Using a protocol. • Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. • Bug reports that need to be addressed and corrections. Playground = Instantiation of Platform Crowdsourcing

  6. Claims • Protocol. Defines scientific discourse. • Scholars make a prediction about their performance in protocol. • Predicate that decides whether refutation is successful. Refutation protocol collects data for predicate. • As a starter: Think of a claim as a mathematical statement: EA or AE. • all planar graphs have a 4 coloring. Crowdsourcing

  7. Benefits Return On Investment for playground designers: a small investment in defining a playground (Domain=(Instance,Solution,valid,quality), Claim=(Protocol, etc.)) produces an interactive environment to assimilate and create domain knowledge. Crowdsourcing

  8. Benefits • Return on Investment for scholars and avatar designers: The SCG rules need to be learned only once because they are the same across playgrounds. A small investment in learning the SCG rules and a domain etc. leads to numerous learning and teaching and innovation opportunities. The more a scholar teaches, the higher the scholar’s reputation. Crowdsourcing

  9. Global Warming • Alice’ Claim: The earth is warming significantly. • Refutation protocol: Bob tries to refute. • Alice must provide a data set DS satisfying a property defined precisely by the refutation protocol. • Bob applies one of the allowed analysis methods M defined precisely by the refutation protocol. • Bob wins iff M(DS) holds. Crowdsourcing

  10. Independent Set • Protocol / claim: At Least As Good • Bob provides undirected graph G. • Bob computes independent set sB for G (secret). • Alice computes independent set sA for G. • Alice wins, if size(sA) >= size(sB). Crowdsourcing

  11. Overview • Organizational problem that SCG solves • What is SCG in detail? • Crowdsourcing • Formal Properties of SCG • Applications • Disadvantages • Conclusions Crowdsourcing

  12. What is SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge Crowdsourcing

  13. A Virtual WorldAvatar’s View Avatar does complex work Claims, Instances, Solutions Opponents’ communication, Feedback Administrator does simple checking (usually efficient) Results • Agreed Claims: statements about algorithms • = Social welfare • Algorithms in Avatar Crowdsourcing

  14. Life of an avatar: (propose+ (oppose |agree)+ provide* solve*)* Avatars propose and (oppose|agree) proposed claims egoistic Alice egoistic Bob CB1 opposes (1) CA1 CB2 provides instance (2) CA2 solves instance LOSES WINS! not as well as she expected based on CA2 (3) CA3 reputation 10 reputation 1000 transfer 200 CA4 Crowdsourcing

  15. What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? Crowdsourcing

  16. blamed decisions: propose(Alice,c) refute(A,B,c) strengthen(Alice,Bob,c,cs) agree(A,B,c) Essence of Game Rules • actors: • proposer=verifier (1. arg to refute, usually Alice), • opposer=falsifier (2. arg to refute, usually Bob) • LifeOfClaim(c) = propose(Alice,c) followed by (oppose(Alice,Bob,c)|agree(Alice,Bob,c)). • oppose(Alice,Bob,c) = (refute(Alice,Bob,c)|strengthen(Alice,Bob,c,cs)), where stronger(c,cs). • strengthen(Alice,Bob,c,cs) = !refute(Bob,Alice,cs). • agree(Alice,Bob,c) = !refute(Alice,Bob,c) and !refute(Bob,Alice,c) and refute(Alice,Bob,!c) and refute(Bob,Alice,!c) Crowdsourcing

  17. Winning/Losing • propose(Alice,c), refutationTry(Alice,Bob,c) • If Alice first violates a game rule, Bob is the winner. • If Bob first violates a game rule, Alice is the winner. • If none violate a game rule: the claim predicate c.p(Alice,Bob,in,out) decides. Crowdsourcing

  18. Game Rules for Playground • legal(in) • legal(out) • valid(in,out) • belongsTo(in, instanceSet) • each move must be within time-limit Crowdsourcing

  19. Protocol Language ProtocolSpec = <steps> List(Step). Step = <action> Action "from" <role> Role. interface Role = Alice | Bob. Alice = "Alice". Bob = "Bob". interface Action = ProvideAction | SolveAction. ProvideAction = "instance". // solve the instance provided in // step # stepNo. // stepNo is 0-based. SolveAction = "solution" "of" <stepNo> int. Crowdsourcing

  20. How to achieve loosely coupled collaboration? • Information exchange is based on values. Knowledge how to produce values is secret. • Assign blame correctly to Alice or Bob based on outcome of refutation protocol. • Every claim has a negation (using the idea of Hintikka’s dual game). Crowdsourcing

  21. Dual Game / Negation • Each game G has a dual game which is the same as G except that the players ∀ and ∃ are transposed in both the rules for playing and the rules for winning. The game G(¬φ) is the dual of G(φ). Crowdsourcing

  22. A claim is • Meta information about one’s performance when interacting with another clever being. • Meta information about the performance of one’s program. Crowdsourcing

  23. How is collaboration working? • Scholars make claim about their performance in a given context. • Scholars make claim about the performance of their avatar in a given context. • Opponent finds input in context that contradicts claim. Claim is refuted. Crowdsourcing

  24. Playground Design • Define several languages • Instance • Solution • Claim • InstanceSet • Define protocol or reuse existing protocol. • Implement interfaces for corresponding classes. Crowdsourcing

  25. Who are the scholars? • Students in a class room • High school • University • Members of the Gig Economy • Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. • Potential employees • Anyone with web access; Intelligent crowd. Crowdsourcing

  26. How to engage scholars? • Several binary games between Alice and Bob. • Alice must propose C or !C for one of the allowed C. • Bob must agree with or oppose what Alice proposes. • Agree(C) • Bob defends C against Alice. • Bob refutes !C against Alice. • Alice defends C against Bob. • Alice refutes !C against Bob. Crowdsourcing

  27. How to engage scholars?Opposition • Central to opposition is refutation. • Claim defined by protocol. • Simplest protocol: • Alice provides Input in. • Bob computes Output out: valid(in,out) • Alice defends if quality(in,out)<q. • Bob refutes if quality(in,out)>=q. • Claims: C(q), q in [0,1]. Crowdsourcing

  28. Overview • Organizational problem solved by SCG • What is SCG in detail? • Crowdsourcing • Formal Properties of SCG • Applications • Disadvantages • Conclusions Crowdsourcing

  29. Crowdsourcing • Active area: Recent Communication of the ACM article. • Wikipedia, FoldIt, TopCoder, … • We want a family of crowdsourcing systems with provable properties. Crowdsourcing

  30. Crowdsourcing Platform • Crowdsourcing • is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call. • enlists a crowd of humans to help solve a problem defined by the system owners. • A crowdsourcing platform is a generic tool that makes it easy to develop a crowdsourcing system. Crowdsourcing

  31. Crowdsourcing Platform • The job, target problem is • to solve instances of a problem and make claims about the solution process. • to build knowledge base of claims and techniques to defend the claims Crowdsourcing

  32. Requirements for Crowdsourcing Platform • Find a good way to combine user contributions to solve the target problem. • Find a good way to evaluate users and their contributions. • Find a good way to recruit and retain users. Crowdsourcing

  33. SCG Court • Web application • Software developers register with SCG Court and choose playgrounds they want to compete in. • They register their avatars in the appropriate playgrounds in time for the next tournament. • Avatars get improved between tournaments based on ranking achieved and game history. Crowdsourcing

  34. Combine user contributions • Users build on each others work: strengthening and checking. • Users check each others claims for correct judgment. • Claims are defended and refuted. • Users trade reputation for information. Crowdsourcing

  35. Learning cycle • Alice wins reputation with claim c because Bob made a wrong decision • Alice gives information about artifact related to c. Alice teaches Bob. • Bob integrates information into his know-how. Bob learns from Alice. • Bob hopefully has learned enough and will no longer make a wrong decision about c. Crowdsourcing

  36. Voting with Justification • I vote • for this claim (agree) because I can defend it and refute its negation. • against this claim because I can oppose it (refute or strengthen). Crowdsourcing

  37. Evaluate users and their contributions • Calculate reputation • confidence by the proposer that a claim is good (gc) • confidence by the opposer (refute or strengthen) that the claim is bad (bc) • The scholars are encouraged to set their confidences truthfully. Otherwise they don't gain enough reputation or they lose too much reputation. Crowdsourcing

  38. Reputation Update up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose up: reputation goes up, but has to provide knowledge that might reveal secret technique. down: reputation goes down, but might gain knowledge that reveals secret technique. Crowdsourcing

  39. Reputation Update up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose • confidence: • proposer: claim is good: gc • opposer: claim is bad: bc • r = result of reputation protocol. • Reputation update: r*gc*bc (various refinements are possible) Crowdsourcing

  40. Perfect • Being perfect means to make perfect decisions. • up: if you are perfect, you will not lose. • down: if the other is perfect, you will not win. up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose Crowdsourcing

  41. Overview • Organizational problem solved by SCG • What is SCG in detail? • Crowdsourcing • Formal Properties of SCG • Applications • Disadvantages • Conclusions Crowdsourcing

  42. Formal Properties of SCG • Soundness: • Only false claims are refuted. • Only true claims are defended. • SCG is not sound because it adapts to the skill level of the scholars. E.g., • Alice proposes a false claim and still defend it, because Alice and Bob are weak, or • Alice proposes a true claim and not defend it, because Alice is weak. • We want to prove formal properties that don’t imply soundness. Crowdsourcing

  43. Formal Properties • Properties • Community Property • Equilibrium • Convergence • Assumption: claims are bivalent (true or false); disallow indeterminate claims. Crowdsourcing

  44. For every faulty decision action there exists an exposing reaction. • decision propose(A,c): if c is not true, refute(A,B,c) or strengthen(A,B,c,cs) expose. • decision oppose(Alice,Bob,c)|agree(Alice,Bob,c): • if Bob decides to oppose but does not oppose successfully, his oppose action is blamed. Bob discouraged to attack without good reason. • if Bob decides to agree but does not agree successfully, his agree action is blamed. Crowdsourcing

  45. Community Property • For every faulty decision action there exists an exposing reaction that blames the bad decision. • Reasons: • We want the system to be egalitarian. • It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. • Faulty decisions must be exposable. It may take effort. Crowdsourcing

  46. Community PropertyAlternative formulation • If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob. • if Alice is perfect, there is no chance of losing. • If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob. • egalitarian game Crowdsourcing

  47. Summary: faulty decisions • propose(Alice,c),c=false • propose(Alice,c),c=not optimum, c=true • refute(Alice,Bob,c),c=true • strengthen(Alice,Bob,c,cs),c=optimum • strengthen(Alice,Bob,c,cs),c=false • agree(Alice,Bob,c),c=false • agree(Alice,Bob,c),c=not optimum, c=true Crowdsourcing

  48. Community PropertyCase 1 • Alice’ decision propose(Alice,c) proposes claim c as a claim that is true. Let’s assume c is false. Alice introduced a fault into the knowledge base. • There must be a reaction that assigns blame to Alice’ decision. Here it is: Bob decides to oppose: oppose(Alice,Bob,c), specifically to refute: refute(Alice,Bob,c). There must be a successful refutation. 1. propose(Alice,c),c=false Crowdsourcing

  49. Community PropertyCase 2 • Alice’ decision propose(Alice,c) proposes claim c as a claim that is optimum. Let’s assume c is not optimum, but true, and can be strengthened. Alice introduced a fault into the knowledge base. • There must be a reaction that assigns blame to Alice’ decision. Here it is: Bob decides to oppose: oppose(Alice,Bob,c), specifically to strengthen: strengthen(Alice,Bob,c,cs). There must be a choice for cs so that refute(Bob,Alice,cs) returns false, independent of Alice’ strategy. 2. propose(Alice,c),c=not optimum, c=true Crowdsourcing

  50. Community PropertyCase 3 • Bob’s decision refute(Alice,Bob,c) is wrong, if c is true. Bob tries to introduce a fault into the knowledge base. • There must be a reaction by Alice that assigns blame to Bob’ decision to refute. Because c is true, there must be a defense of c by Alice, i.e., refute(Alice,Bob,c) returns false independent of Bob’s strategy. Bob’s decision to refute is blamed. 3. refute(Alice,Bob,c),c=true Crowdsourcing

More Related