A Popperian Platform for Programming and Teaching the Global Brain

1. A Popperian Platform for Programming and Teaching the Global Brain Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston

2. Outline • Introduction Introduction Introduction • Theoretical background Theory & hypotheses Theory • Methods for playground design Methods Methods • Results Results & analysis Results • Conclusions Conclusion Conclusion

3. Conclusion Introduction Theory Methods Results Results SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, … • SCG usage for teaching • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but for competitive innovation) • Theoretical Properties of SCG

4. Conclusion Introduction Theory Methods Results What SCG helps with • How to identify experts? • How to decide if an answer is worthwhile? • Use scholars to choose the winners • How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. • The scholars try to reverse engineer the solutions of winning scholars.

5. Claims • Protocol. Defines scientific discourse. • Scholars make a prediction about their performance in protocol. • Predicate that decides whether refutation is successful. Refutation protocol collects data for predicate. • As a starter: Think of a claim as a mathematical statement: EA or AE. • all planar graphs have a 4 coloring. Crowdsourcing

6. Who are the scholars? • Students in a class room • High school • University • Members of the Gig Economy • Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. • Potential employees • Anyone with web access; Intelligent crowd. Crowdsourcing

7. Conclusion Introduction Theory Methods Results What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? Crowdsourcing

8. Conclusion Introduction Theory Methods Results Degree of automation with SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge Crowdsourcing

9. happy = no scholar is ignored. Organizational Problem Solved • How to design a happy scientific community that encourages its members to really contribute. • Control of scientific community • tunable SCG rules • Specific domain, claim definition to narrow scope. Crowdsourcing

10. Conclusion Introduction Theory Methods Results What is a loose collaboration? • Scholars can work independently on an aspect of the same problem. • Problem = decide which claims in playground to oppose or agree with. • How is know-how combined? Using a protocol. • Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. • Bug reports that need to be addressed and corrections. Playground = Instantiation of Platform Crowdsourcing

11. Theory • Extensive Form Representation of Game • Community Property: All faulty actions can be exposed.

12. Conclusion Introduction Theory Methods Results Extensive-form representation • the players of a game: 1 and 2 • for every player every opportunity they have to move • what each player can do at each of their moves • what each player knows for every move • the payoffs received by every player for every possible combination of moves

13. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) u:1 2 s:1 2 refute(C,2,1) p(C, …)?(1,-1):(-1,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1) s:1 2 u:1 2 s:1 2 u:1 2

14. Refutation Protocol • Collects data given to predicate p. Alternates. refute(C,proposer,other) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. p(C, …)?(1,-1):(-1,1) claim payoff for proposer if p true (defense) payoff for other if p true (defense) payoff for proposer if p false (refutation) payoff for other if p false (refutation)

15. Reinterpret Refutation • Refutation leads to successful strengthening or successful agreement.

16. blamed decisions: propose(1,C) refute(1,2,C) strengthen(1,2,C,C’) agree(1,2,c) Essence of Game Ruleswithout Payoff • actors: 1, 2 • LifeOfClaim(C) = propose(1,C) followed by (oppose(1,2,C)|agree(1,2,C)). • oppose(1,2,C) = (refute(1,2,C)|strengthen(1,2,C,C’)), where stronger(C,C’). • strengthen(1,2,C,C’) = !refute(2,1,C’). • agree(1,2,C) = !refute(2,1,C) Crowdsourcing

17. Winning/Losing • Scholar who first violates a game rule, loses. • If none violate a game rule: the claim predicate c.p(1,2, …) decides. Crowdsourcing

18. Game Rules for Playground • All objects exchanged during protocol must be legal and valid. • Each move must be within time-limit. Crowdsourcing

19. Example: Independent Set • Alice = proposer, Bob = other. • Protocol / claim: AtLeastAsGood. Alice claims to be at least as good as Bob at IS. • Bob provides undirected graph G. • Bob computes independent set sB for G (secret). • Alice computes independent set sA for G. • Alice wins, if size(sA) >= size(sB) (= p(sA,sB)). Crowdsourcing

20. Conclusion Introduction Theory Methods Results More examples of Protocols • Let f(x,y)=x*y+(1-x)(1-y^2)). Alice claims Math(0.61): Bob constructs an x in [0,1] and Alice construct a y in [0,1], and Alice guarantees that f(x,y)> 0.61. True claim but can be strengthened to 0.618. • Alice claims Solar(RawMaterials,m,0.61). Bob constructs raw materials r in RawMaterials and Alice constructs a solar cell s in Solution from r using money m and so that efficiency(s)> 0.61.

21. Conclusion Introduction Theory Methods Results Community Property • For every faulty decision action there exists an exposing reaction that blames the bad decision. • Reasons: • We want the system to be egalitarian. • It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. • Faulty decisions must be exposable. It may take effort. Crowdsourcing

22. Community PropertyAlternative formulation • If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob. • if Alice is perfect, there is no chance of losing. • If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob. • egalitarian game Crowdsourcing

23. Summary: faulty decisions • propose(Alice,C),C=false • propose(Alice,C),C=not optimum, C=true • refute(Alice,Bob,C),C=true • strengthen(Alice,Bob,c,cs),c=optimum • strengthen(Alice,Bob,c,cs),c=false • agree(Alice,Bob,c),c=false • agree(Alice,Bob,c),c=not optimum, c=true Crowdsourcing

24. Conclusion Introduction Theory Methods Results SCG Equilibrium • Reputations of scholars are stable. • The science does not progress; bugs are not fixed, no new ideas are introduced. • Extreme, desirable situation: All scholars are perfect: they propose optimal claims that can neither be strengthened nor refuted. Crowdsourcing

25. Claims: convergence to optimum over strengthening 1 false claims (refutable) correct valuation quality strengthening true claims (defendable) Crowdsourcing 25 0

26. Conclusion Introduction Theory Methods Results Convergence • if every faulty action is exposed, convergence is guaranteed. Crowdsourcing

27. Conclusion Introduction Theory Methods Results Methods • Developed Platform SCG Court = Generator of teaching/innovation playgrounds • http://sourceforge.net/p/generic-scg/code-0/11 0/tree/GenericSCG/ • Developed Algorithms Course using Piazza based on platform experience Crowdsourcing

28. Avatar Interface • AvatarI • public List<Claim> propose(List<Claim> forbiddenClaims); • public List<OpposeAction> oppose(List<Claim> claimsToBeOpposed); • public InstanceI provide(Claim claimToBeProvided); • public SolutionI solve(SolveRequest solveRequest);

29. Instance Interface • InstanceI • boolean valid(SolutionI solution, Config config); • double quality(SolutionI solution);

30. InstanceSet Interface • InstanceSetI • Option<String> belongsTo(InstanceI instance); • Option<String> valid(Config config); }}

31. Protocol Interface • ProtocolI • double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); • ProtocolSpec getProtocolSpec(); • boolean strengthenP(Claim oldClaim, Claim strengthenedClaim);

32. Claim Class, for all playgrounds • Claim • public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence)

33. Protocol Library • AsGoodAsYou.java • ExistsForAll.java • ForAllExists.java • Renaissance.java • Survivor.java

34. Piazza

35. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 High competition refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) u:1 2 s:1 2 refute(C,2,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1) s:1 2 u:1 2 s:1 2 u:1 2

36. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 Low competition refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) s: successful u: unsuccessful propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)? (0,0) :(0,1) refute(C’,2,1) u:1 2 s:1 2 refute(C,2,1) p(C, …)?(0,0): (1,0) p(C’, …)?(0,1): (0,0) s:1 2 u:1 2 s:1 2 u:1 2

37. Competition Knob: minimum • For each scholar • count claims that were successfully opposed (refuted or strengthened) • encourages strong claims • gather information from competitors for free • count claims that were not successfully agreed • Good for teaching • students want minimum competition • good students want to build social capital and help weaker students

38. Piazza Results • Lower competition knob for teaching. • For optimization claims got significant scientific discourse. • Playgrounds cannot have too many scholars, otherwise they are overwhelmed. • about 5 is a good size • use hierarchical playgrounds: winning teams compete again

39. Piazza Results • Do not give hints at solutions. This significantly decreased the amount of discourse taking place.

40. Conclusions • Transition • refute: (1,-1):(-1,1) -> (0,0) :(0,1) • strengthen: (-1,1):(1,-1) -> (0,1): (0,0) • agree: (0,0):(1,-1) -> (0,0): (1,0) • creates a better playground for learning by lowering competition and increasing teaching between scholars.

41. Conclusions • Flexible use of SCG using a forum environment with threads and replies using optimization optimization playgrounds is productive: • teams took turns leapfrogging each other

42. Conclusion Introduction Theory Methods Results Transformation: performance debate • Diverse governance modes • Liberalization • Underperformance - legitimacy • Mixed results of privatization • Challenges for state Public Output Input Public-private Private