A Popperian Platform for Programming and Teaching the Global Brain

1. A Popperian Platform for Programming and Teaching the Global Brain Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston

2. A claim is … • information about one’s performance when interacting with another clever being in a specific domain. • information about the performance of one’s program. Crowdsourcing

3. Outline • Introduction Introduction • Theoretical Background Theory • Methods of Exploration Methods • Results Results • Conclusions and Future Work Conclusion

4. Conclusion Introduction Theory Methods Results Introduction SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, Renaissance, … • Make SCG a part of cyber-infrastructure (e-science) to support teaching and innovation in constructive domains. • SCG usage for teaching • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but for competitive innovation) • Theoretical Properties of SCG

5. Conclusion Introduction Theory Methods Results Specker • Claims: • Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c • Example 1 • X = Conjunctive Normal Forms with various restrictions • Y(X) = Assignments to CNFs • f(x,y) = fraction of satisfied clauses in x under y • c in [0,1], e.g., c= 0.61 • Example 2 • X = [0,1] • Y(X) = [0,1] • f(x,y)=x*y+(1-x)(1-y^2)) • c in [0,1], e.g., c=0.61

6. Conclusion Introduction Theory Methods Results Popper • One of the philosophers of science who has had the biggest impact. • Popper’s solution: Falsificationism: A claim is falsifiable if you can imagine an observation that would cause you to reject the theory. • That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it.

7. Conclusion Introduction Theory Methods Results What SCG helps with • How to identify experts? • How to decide if an answer is worthwhile? • Use scholars to choose the winners • How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. • The scholars try to reverse engineer the solutions of winning scholars.

8. Claim involving Experiment Claim ExperimentalTechnique(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. Crowdsourcing

9. Who are the scholars? • Scientists • Students in a class room • High school • University • Members of the Gig Economy • Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. • Potential employees (Facebookon kaggle.com) • Anyone with web access; Intelligent crowd.

10. Kaggle.com Competitions2012 • Facebook recruiting competitions • Task: Data scientist • Reward: Job • Teams: 197 • Heritage Health Prize • Task: Hospital admissions • Reward: $ 3 million • Teams: 1118 • Chess ratings – Elo versus the Rest of the World • Task: Predict outcome of chess games • Reward: $ 617 • Teams: 257

11. Kaggle.com Competitions2012 • Eye Movements Verification and Identification • Task: Identify people • Reward: Kudos • Teams: 51 • EMC Data Science Global Hackathon • Task: Air Quality Prediction • Reward $ 7030 • Teams: 114

12. Conclusion Introduction Theory Methods Results What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? • Scholars are free to invent; game rules don’t limit creativity!

13. Conclusion Introduction Theory Methods Results Degree of automation with SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge

14. happy = can contribute, not ignored, can be creative, can thrive, have opportunity to learn Organizational Problem Solved • How to design a happy scientific community that encourages its members to really contribute. • Control of scientific community • tunable SCG rules • Specific domain, claim definition to narrow scope.

15. Conclusion Introduction Theory Methods Results What is a loose collaboration? • Scholars can work independently on an aspect of the same problem. • Problem = decide which claims in playground to oppose or agree with. • How is know-how combined? Using a protocol. • Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. • Bug reports that need to be addressed and corrections. Playground = Instantiation of Platform

16. Conclusion Introduction Theory Methods Results Theory • Extensive Form Representation of Game • Community Property: All faulty actions can be exposed. • SCG Equilibrium • Convergence to optimum claim

17. Conclusion Introduction Theory Methods Results Extensive-form representation • the players of a game: 1 and 2 • for every player every opportunity they have to move • what each player can do at each of their moves • what each player knows for every move • the payoffs received by every player for every possible combination of moves

18. Large Action Spaces • Thick arrows mean: select from a usually large number of choices 1 2

19. Conclusion Introduction Theory Methods Results Refutation Protocol • Collects data given to predicate p. Alternates. refute(C,proposer,other) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. p(C, …)?(1,-1):(-1,1) claim payoff for proposer if p true (defense) payoff for other if p true (defense) payoff for proposer if p false (refutation) payoff for other if p false (refutation)

20. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(1,-1):(-1,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1)

21. Reinterpret Refutation • Refutation leads to successful strengthening or successful agreement.

22. Conclusion Introduction Theory Methods Results Game Rules for Playground • All objects exchanged during protocol must be legal and valid. • Each move must be within time-limit. • Scholar who first violates a playground rule, loses.

23. Example: Independent Set • Alice = proposer, Bob = other. • Protocol / claim: AtLeastAsGood. Alice claims to be at least as good as Bob at IS. • Bob provides undirected graph G. • Bob computes independent set sB for G (secret). • Alice computes independent set sA for G. • Alice wins, if size(sA) >= size(sB) (= p(sA,sB)).

24. Conclusion Introduction Theory Methods Results Logic with Soundness claims sentences good bad not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum Crowdsourcing

25. Conclusion Introduction Theory Methods Results Scientific Community Game Logic with Community Principle claims sentences good bad disagreed by two scholars agreed by two scholars there exists two-party certificate to expose misclassification Crowdsourcing

26. Comparison Logic and SCG Logic Scientific Community Game sentences = claims good bad evidence for goodness defense, checkable uncertainty of defense evidence for badness refutation, checkable uncertainty of refutation Personified sentences • sentences • true • false • proof for being true • proof system, checkable • guaranteed defense • proof for being false • proof system, checkable • guaranteed refutation • Universal sentences Crowdsourcing

27. Conclusion Introduction Theory Methods Results Community Property • For every faulty decision action there exists an exposing reaction that blames the bad decision. • Reasons: • We want the system to be egalitarian. • It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. • Faulty decisions must be exposable. It may take effort.

28. Community PropertyAlternative formulation • If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob. • if Alice is perfect, there is no chance of losing. • If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob. • egalitarian game

29. Summary: faulty decisions • propose(Alice,C),C=false • propose(Alice,C),C=not optimum, C=true • refute(Alice,Bob,C),C=true • strengthen(Alice,Bob,c,cs),c=optimum • strengthen(Alice,Bob,c,cs),c=false • agree(Alice,Bob,c),c=false • agree(Alice,Bob,c),c=not optimum, c=true

30. Conclusion Introduction Theory Methods Results SCG Equilibrium • Points (reputations) of scholars are stable. • The science does not progress; bugs are not fixed, no new ideas are introduced. • Extreme, desirable situation: All scholars are perfect: they propose optimal claims that can neither be strengthened nor refuted.

31. Claims: convergence to optimum over strengthening 1 false claims (refutable) correct valuation quality strengthening true claims (defendable) 31 0

32. Conclusion Introduction Theory Methods Results Convergence • Given a family of claims totally ordered with respect to strengthening and containing an optimum claim. If every faulty action is exposed, convergence to the optimum claim is guaranteed by using a search strategy such as binary search.

33. Conclusion Introduction Theory Methods Results Methods of Exploration • Developed Platform SCG Court = Generator of teaching/innovation playgrounds • http://sourceforge.net/p/generic-scg/code-0/11 0/tree/GenericSCG/ • Developed numerous playgrounds for avatars. • Developed Algorithms Course using Piazza based on SCG Court experience • role of scholar played by humans • piazza.com: encourages students to answer each other’s questions.

34. Conclusion Introduction Theory Methods Results Avatar Interface • AvatarI • public List<Claim> propose(List<Claim> forbiddenClaims); • public List<OpposeAction> oppose(List<Claim> claimsToBeOpposed); • public InstanceI provide(Claim claimToBeProvided); • public SolutionI solve(SolveRequest solveRequest); • from http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/src/scg/scg.beh

35. Conclusion Introduction Theory Methods Results Instance Interface • InstanceI • boolean valid(SolutionI solution, Config config); • double quality(SolutionI solution);

36. InstanceSet Interface • InstanceSetI • Option<String> belongsTo(InstanceI instance); • Option<String> valid(Config config); }}

37. Protocol Interface • ProtocolI • double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); • ProtocolSpec getProtocolSpec(); • boolean strengthenP(Claim oldClaim, Claim strengthenedClaim);

38. Claim Class, for all playgrounds • Claim • public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence)

39. Conclusion Introduction Theory Methods Results Protocol Library • ExistsForAll.java • ForAllExists.java • Renaissance.java • AsGoodAsYou.java • Survivor.java

40. Conclusion Introduction Theory Methods Results Claim Kinds in SCG Court • Claim C(instance, q) • Claim C(InstanceSet, q) • Claim MaxResource(Algorithm,InstanceSet,n,ResExp) • Claim MinResource(Algorithm,InstanceSet,n,ResExp) • Claim IAmTheBest(), AtLeastAsGoodAsYou() • Claims from predicate logic and some of its generalizations (IF Logic)

41. Piazza: Gale-Shapley • Gale-Shapley (GS) algorithm is fixed. • Alice provides secret instance of size n, solves it with GS (counts number of iterations) and publishes it: Ac(n). • Bob provides secret instance of size n, solves it with GS and publishes it Bc(n). • Alice wins if Ac(n) > Bc(n).

42. Reverse EngineeringPlayground Design Issues • input, output, algorithm unknown • for some problems input-output pairs help to invent the algorithm • input unknown, output, algorithm • e.g., output for worst-case running-time, find input which is worst-case • for some problems finding

43. Conclusion Introduction Theory Methods Results Playground Design • Make sure that the scholars don’t have to reveal their secrets. If they do. • Remember who was first? • Provenance of winning idea.

44. Conclusion Introduction Theory Methods Results SCG with piazza.com • use piazza.com for posting • playground definition by playground designer • for scholars • claims • executions of refutation protocols • Community policing • Used JSON to exchange claims, instances and solutions. • Used links when they did not fit in piazza window.

45. Conclusion Introduction Theory Methods Results JSON example • We define the JSON notation for defining a preference p as follows: • {"n":3, • "manPref" : [[2,1,0],[1,0,2],[0,1,2]], • "womanPref : [[2,1,0],[1,3,2],[3,1,2]] } • Easy to parse in a variety of programming languages and also readable by humans.

46. Piazza ExperienceGale-Shapley • We propose that, for all integers n > 0, the maximum iterations the Gale-Shapely algorithm with n men and n women can produce is n(n-1)+1.Note: Thus far, the algorithm used for all other claims arrives at only (n(n+1))/2.

47. Piazza Experience • Leaf Covering: Improved running time from quadratic to constant time.

48. Conclusion Introduction Theory Methods Results Results SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, http://eterna.cmu.edu/ … • SCG usage for teaching using forum • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but good for competitive innovation) • Theoretical Properties of SCG

49. Competition tuning: minimum • For each scholar • count claims that were successfully opposed (refuted or strengthened) • encourages strong claims • gather information from competitors for free • count claims that were not successfully agreed • Good for teaching • students want minimum competition • good students want to build social capital and help weaker students

50. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 High competition refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1)