Virtual Scientific-Community-Based Foundations for Popperian e-Science

1. Virtual Scientific-Community-Based Foundations for Popperian e-Science Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston

2. inspired by ScienceWISE Ontology Organization Mathematics Computer Science Mechanism Design Mathematical Logic Programming Game Theory MetaGaming ExtensiveForm Socio-Technical System The Global Brain Dialog Games IF Logic

3. A claim is … • information about one’s performance when interacting with another clever being in a specific domain. • information about the performance of one’s program. Crowdsourcing

4. Outline • Introduction Introduction • Theoretical Background Theory • Methods of Exploration Methods • Results Results • Conclusions and Future Work Conclusion

5. Conclusion Introduction Theory Methods Results Introduction SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, Renaissance, … • Make SCG a part of cyber-infrastructure (e-science) to support teaching and innovation in constructive domains. • SCG usage for teaching • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but for competitive innovation) • Theoretical Properties of SCG • Take Home: EASY USE WITH STANDARD TOOLS • group research, teaching, (intelligent) crowd sourcing

6. Conclusion Introduction Theory Methods Results Popper • One of the philosophers of science who has had a big impact. • Popper’s solution: Falsification: A claim is falsifiable if you can imagine an observation that would cause you to reject the claim. • That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it.

7. Conclusion Introduction Theory Methods Results What SCG helps with • Build and maintain knowledge bases (sets of claims believed to be true). • How to identify experts? • How to decide if an answer is worthwhile? • Use scholars to choose the winners • How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. • The scholars try to reverse engineer the solutions of winning scholars.

8. Abstraction from 4 Examples • From a CS journal paper • Insilico experiment • From kaggle.com: Facebook competition • From a calculus problem

9. Example 1: From an Abstract of a 2005 Journal Paper • An instance of a constraint satisfaction problem (CSP) is variable k-consistent if any subinstance with at most k variables has a solution. • For a fixed constraint language L, r(k,L) is the largest ratio such that any variable k-consistent instance has a solution that satisfies at least a fraction of r(k,L) of the constraints.

10. Example 1 • From a 2005 TCS paper: Locally Consistent Constraint Satisfaction Problems by Manuel Bodirsky and Daniel Kral. • Example • L = CNF • k = 1 • What is r(1,CNF)? • Claims: r(1,CNF) = 0.6, r(1,CNF) = 0.7

11. Example 1: Making a game to determine r(1,CNF) • Observation: claims are falsifiable playing a two person game.

12. Example 2: Claim involving Insilico Experiment Claim InsilicoExperimental(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. Crowdsourcing

13. Example 2: Making a game to determine InsilicoExperimental(X,Y,q,r) • Observation: claims are falsifiable playing a two person game.

14. Conclusion Introduction Theory Methods Results Example 3: Data mining • Facebook competition from Kaggle.com: • Given a social network graph x with deleted edges and the original social network graph gs (secret, from a family X of social networks) • guess the complete social network graph y • quality(x, gs, y) = mean average precision (adapted from IR) • I claim I can achieve a mean average precision of q • for social graphs in family X: DM1(X,q) • for a specific reduced social graph: DM2(x,q)

15. Example 3: Making a game to determine the optimal claims • Observation: claims DM1(X,q) and DM2(x,q) are falsifiable playing a two person game.

16. Conclusion Introduction Theory Methods Results Example 4: Specker • Claims: • Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c • Example 1 • X = Conjunctive Normal Forms with various restrictions • Y(X) = Assignments to CNFs • f(x,y) = fraction of satisfied clauses in x under y • c in [0,1], e.g., c= 0.61 • Example 2 (a reduction of example 1) • X = [0,1] • Y(X) = [0,1] • f(x,y)=x*y+(1-x)(1-y^2)) • c in [0,1], e.g., c=0.61

17. Example 4: Specker • Observation: claims Specker(X,Y,f,c) are falsifiable playing a two person game.

18. What is the abstraction? • Sets of claims • Claims are falsifiable • …

19. Playgrounds • Each playground defines: • domain • claims language • specific protocol • data exchanged • configuration data PG1 PG2 SC2 SC4 claims C21 C22 C23 … claims C11 C12 C13 … SC1 SC3 D1 SC5 SC1 RP1 RP2 D2 SC1 SC1 • SCG defines: • refutation protocol interface • generic rules for all playgrounds

20. Example 1: Making a game to determine r(1,CNF) • Observation: claims are falsifiable playing a two person game. defendable = !refutable • propose r(1,CNF) = 0.7 • refutable • propose r(1,CNF) = 0.6 can be strengthened to r(1,CNF) = 0.61 which is defendable (refutation attempts will be unsuccessful) • propose r(1,CNF) = (sqrt(5)-1)/2 ~ 0.618 … optimum: defendable and cannot be strengthened

21. Who are the scholars? • Scientists • Students in a class room • High school • University • Members of the Gig Economy • Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. • Potential employees (Facebookon kaggle.com) • Anyone with web access; Intelligent crowd.

22. Kaggle.com Competitions2012 • Facebook recruiting competitions • Task: Data scientist • Reward: Job • Teams: 197 • Heritage Health Prize • Task: Hospital admissions • Reward: $ 3 million • Teams: 1118 • Chess ratings – Elo versus the Rest of the World • Task: Predict outcome of chess games • Reward: $ 617 • Teams: 257

23. Kaggle.com Competitions2012 • Eye Movements Verification and Identification • Task: Identify people • Reward: Kudos • Teams: 51 • EMC Data Science Global Hackathon • Task: Air Quality Prediction • Reward $ 7030 • Teams: 114

24. Conclusion Introduction Theory Methods Results What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? • Scholars are free to invent; game rules don’t limit creativity!

25. Conclusion Introduction Theory Methods Results Degree of automation with SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge

26. happy = can be creative, can thrive, have opportunity to learn, not ignored Organizational Problem Solved • How to design a happy scientific community that encourages its members to really contribute. • Control of scientific community • tunable SCG rules • Specific domain, claim definition to narrow scope.

27. Playground • defines • what is wanted, e.g., an algorithm S in a particular domain (inputs/outputs) • evaluation, e.g., how S is evaluated (quality) • claims, e.g., what kind of claims can be made about S (expression with quantifiers) • A playground defines WHAT is desired and the scholars/avatars define the HOW.

28. Conclusion Introduction Theory Methods Results Theory • Extensive Form Representation of Game • Properties • Community Property: All faulty actions can be exposed. • SCG Equilibrium • Convergence to optimum claim

29. Conclusion Introduction Theory Methods Results Extensive-form representation • the players of a game: 1 and 2 • for every player every opportunity they have to move • what each player can do at each of their moves • what each player knows for every move • the payoffs received by every player for every possible combination of moves

30. Large Action Spaces • Thick arrows mean: select from a usually large number of choices 1 2

31. Conclusion Introduction Theory Methods Results Refutation Protocol • Collects data given to predicate p. Alternates. refute(C,proposer,other) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. p(C, …)?(1,-1):(-1,1) claim payoff for proposer if p true (defense) payoff for other if p true (defense) payoff for proposer if p false (refutation) payoff for other if p false (refutation)

32. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 SCG Core refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(1,-1):(-1,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1)

33. Conclusion Introduction Theory Methods Results Game Rules for Playground • All objects exchanged during protocol must be legal and valid. • Each move must be within time-limit. • Scholar who first violates a playground rule, loses.

34. Conclusion Introduction Theory Methods Results Logic with Soundness claims sentences good bad not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum Crowdsourcing

35. Conclusion Introduction Theory Methods Results Scientific Community Game Logic with Community Principle claims sentences good bad disagreed by two scholars agreed by two scholars there exists a two-party certificate to expose misclassification Crowdsourcing

36. Comparison Logic and SCG Logic Scientific Community Game sentences = claims good bad evidence for goodness defense, checkable uncertainty of defense evidence for badness refutation, checkable uncertainty of refutation Personified sentences • sentences • true • false • proof for being true • proof system, checkable • guaranteed defense • proof for being false • proof system, checkable • guaranteed refutation • Universal sentences Crowdsourcing

37. Conclusion Introduction Theory Methods Results Community Property • For every faulty decision action there exists an exposing reaction that blames the bad decision. • Reasons: • We want the system to be egalitarian. • It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. • Faulty decisions must be exposable. It may take effort.

38. Conclusion Introduction Theory Methods Results Methods of Exploration • Developed Platform SCG Court = Generator of teaching/innovation playgrounds • http://sourceforge.net/p/generic-scg/code-0/11 0/tree/GenericSCG/ • Developed numerous playgrounds for avatars. • Developed Algorithms Course using Piazza based on SCG Court experience • role of scholar played by humans • piazza.com: encourages students to answer each other’s questions.

39. Conclusion Introduction Theory Methods Results Avatar Interface • AvatarI • public List<Claim> propose(List<Claim> forbiddenClaims); • public List<OpposeAction> oppose(List<Claim> claimsToBeOpposed); • public InstanceI provide(Claim claimToBeProvided); • public SolutionI solve(SolveRequest solveRequest); • from http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/src/scg/scg.beh

40. Conclusion Introduction Theory Methods Results Instance Interface (Domain) • InstanceI • boolean valid(SolutionI solution, Config config); • double quality(SolutionI solution);

41. InstanceSet Interface (Domain) • InstanceSetI • Option<String> belongsTo(InstanceI instance); • Option<String> valid(Config config); }}

42. Protocol Interface • ProtocolI • double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); • ProtocolSpec getProtocolSpec(); • boolean strengthenP(Claim oldClaim, Claim strengthenedClaim);

43. Claim Class, for all playgrounds • Claim • public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence)

44. Conclusion Introduction Theory Methods Results Protocol Library • ExistsForAll.java • ForAllExists.java • Renaissance.java • AsGoodAsYou.java • Survivor.java

45. Second Method: Piazza ExperienceGale-Shapley • We propose that, for all integers n > 0, the maximum iterations the Gale-Shapely algorithm with n men and n women can produce is n(n-1)+1.Note: Thus far, the inputs used for all other claims arrives at only (n(n+1))/2.

46. Piazza Experience • Leaf Covering: Improved running time from quadratic to constant time.

47. Conclusion Introduction Theory Methods Results Results SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, http://eterna.cmu.edu/ … • SCG usage for teaching using forum • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but good for competitive innovation) • Theoretical Properties of SCG

48. Conclusion Introduction Theory Methods Results Competition tuning: minimum • For each scholar • count claims that were successfully opposed (refuted or strengthened) • encourages strong claims • gather information from competitors for free • count claims that were not successfully agreed • Good for teaching • students want minimum competition • good students want to build social capital and help weaker students

49. Conclusion Introduction Theory Methods Results Piazza Results • Do not give hints at solutions. This significantly decreased the amount of discourse taking place.

50. Conclusion Introduction Theory Methods Results Conclusions and Future Work • We propose a systematic gamification of teaching STEM domains: • Design an SCG playground where the winning students demonstrate superior domain knowledge. STEM = Science, Technology, Engineering, and Mathematics