Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA

1. The Scientific Community Game: Education and Innovation Through Survival in a Virtual World of Claims Supported by Novartis • Karl Lieberherr • Northeastern University • College of Computer and Information Science • Boston, MA • joint work with • Ahmed Abdelmeged and Bryan Chadwick

2. Why Scientific Community Game(SCG) • … motives in academic publishing: • desire for recognition and respect from the people one regards as peers, • desire to have impact (on conclusions being reached, on the development of the discipline, etc.), and • desire to participate in significant knowledge-buildingdiscourse. • e.g., Scardamalia, M., & Bereiter, C. (1994) Intro SCG

3. SCG is Bio-inspired • Virtual world of scholars based on natural selection • propose, oppose (refute and strengthen) claims • maximize reputation, weak scholars are removed. • Turn problem-solving software into virtual organisms that fend for themselves and survive in a virtual world inhabited by virtual organisms created by your peers. Intro SCG

4. SCG is a web-based implementation of Karl Popper’s science ideas • One of the greatest philosophers of science of the 20th century. • Falsifiability or refutability is the logical possibility that an assertion could be shown false by a particular observation or physical experiment. • Error elimination (refutation), performs a similar function for science that natural selection performs for biological evolution. from Wikipedia Intro SCG

5. Comparison • Karl Popper: Conjectures and Refutations • Scientific Community Game: Claims and Refutations Intro SCG

6. Recognition in SCG • Scholars build their reputation by proposing and opposing claims, by defending their own claims and refuting or strengthening the claims of others. • The higher their reputation, the more recognition. Intro SCG

7. Impact in SCG • Second-order environment • what one scholar does in adapting, changes the environment so that others must readapt. • Developing novel techniques to find superior solutions, challenges others to catch up. Intro SCG

8. Knowledge-Building Discoursein SCG • Communication or debate. • Refutation protocol defines the structure of the debate and who wins. Claims are defined through a refutation protocol. • Knowledge-building: • claims that have been defended predominantly are candidates for truth • claims that have been refuted predominantly are probably false. Intro SCG

9. Goals of SCG • Put knowledge-building discourse on the web giving participants the option to gain recognition and to have impact. • Focus the discourse through precise definition of claims with refutation protocols. • Make knowledge building discourse fun and educational from the high school to the advanced research level. SCG = Scientific Community Game = Specker Challenge Game Intro SCG

10. What do we mean by science? • Science consists of the formulation and testing of hypotheses based on observational evidence. • Ours: Science consists of the formulation and testing of constructive claims based on observational evidence. Construction is computable. Intro SCG

11. What do we mean by Scientific Method • Hypothetico-deductive method: Formulate a hypothesis in a form that could conceivably be falsified by a test on observable data. • Ours: Formulate a constructive claim in a form that could conceivably be falsified by a test using a protocol. The refutation protocol is part of the claim to make very explicit when refutation is successful. Intro SCG

12. SCG claim examples • SCG Claim • AlgorithmicClaim • solve problems of kind D with quality q and resource r • have polynomial time algorithm to solve problems of kind D with quality q • MathematicalClaim • for all x in X exists y in Y: predicate(x,y) • SoftwareClaim • solve problems of kind D with maintainability m • you cannot break into a system of kind D using resource r

13. SCG claim examples • FinancialClaim • if you pay me k dollars (option premium) today, I will promise to buy q shares of stock S up to day d at price p (strike price). Purpose: insurance. • ExperimentalClaim • If I am given raw materials x in X, I can produce product y in Y of quality q and using resources at most r.

14. Tartaglia against Fior 1535 Tartaglia was famed for his algebraic solution of cubic equations which was published in Cardan's Ars Magna. Intro SCG

15. Outline • Introduction • Popper Science, Renaissance History: Tartaglia and Fior • Definition of SCG • Example (Highest safe rung) • Applications: Teaching, Software Development, Research • Claims with secrets and other protocol variants • Output of SCG, Equilibrium • Advantages and Disadvantages • Conclusions Intro SCG

16. Definition of SCG: Domain • Problem: Set • Solution: Set • valid: relation(Problem, Solution) • quality: function(Problem, Solution)->[0..1] Intro SCG

17. Claim(Domain) makes predictions about the future • Problems: Powerset(Domain.Problem) • q: Quality = [0,1] • r: Resource = N+ = positive integer Alice claims to have a technique to solve problems in Problems with at least quality q and using at most resources r. Intro SCG

18. Implied Protocol of Claim(Domain) • Alice claims (problems,q,r), Bob refutes • Bob provides problem prob in Claim.Problems. • Alice solves problem prob providing sol in Domain.Solution. • check: valid(prob,sol) and quality(prob,sol)>=q and sol.resource<=r. • sol.resource returns Alice’ resource consumption to solve problem prob. Karl Popper: Only hypotheses capable of clashing with observation reports are allowed to count as scientific. Intro SCG

19. Claim • Problems: subset of problems • quality in [0,1] 1 quality (how well problems in Problems can be solved) Intro SCG 0

20. Claim over strengthening 1 correct valuation quality strengthening Intro SCG 20 0

21. Bio-inspired computing: Virtual World of SCG-Avatar • SCG-Avatar (Claim(Domain)) • State: Reputation = positive rational number • Activity • propose new claims • oppose claims of others • refute claim(Problems, q, r) • strengthen claim(Problems, q’, r’), q’>q or r’<r • Reputation gain: refute others’ claims and defend own claims (counter refutation attempts) • Reputation loss: unsuccessful refutation of other’s claim and refutation of own claims Intro SCG

22. Tournament • round-robin • Swiss-style • elimination • single • double Intro SCG

23. Summary of SCG Definitions Domain Problem Solution valid(Problem, Solution) quality(Problem, Solution) →[0,1] Claim(Domain) Problems: PowerSet(Domain.Problem) q: Quality = [0,1] r: Resource = N+ Rules of the Scientific Community: propose and oppose, be an active scholar, rules for reputation accumulation. Tournaments Intro SCG

24. Highest Safe Rung • You are doing stress-testing on various models of glass jars to determine the height from which they can be dropped and still not break. The setup for this experiment, on a particular type of jar, is as follows. Intro SCG

25. Highest Safe Rung Bob Alice You have a ladder with n rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung. You have a fixed ``budget'' of k > 0 jars. Only two identical bottles to determine highest safe rung Intro SCG

26. Highest Safe Rung Bob Alice HSR(9,2)≤4 I doubt it: refutation attempt! Alice constructs decision tree T of depth 4 and gives it to Bob. He checks whether T is valid. Bob wins if he finds a flaw. Only two identical bottles to determine highest safe rung Intro SCG

27. x Highest Safe Rung Decision Tree HSR(10,2)=5 no 3 yes y z 1 6 u highest safe rung 0 2 4 9 1 2 3 5 7 9 4 5 8 6 7 8 Intro SCG

28. Formal: HSR • Domain: • Problem: (n,k), k <= n. • Solution: Decision tree to determine highest safe rung. • quality(problem, solution): depth of decision tree / number of rungs • valid(problem, solution): at most k left branches, ... Intro SCG

29. Formal: HSR • Claim(Domain): • Alice claims ({(25,2)},9/25,5 seconds) • {(25,2)}: set of problems (singleton) • 9/25: quality • 5 seconds: resource • Refutation Protocol: • Bob refutes: only one problem: (25,2) • Alice: solves problem by providing decision tree t. • predicate: t is a valid decision tree for (25,2) of depth 9 Intro SCG

30. SCG(HSR) Karl Lieberherr SCG(HSR)

31. Overview • Showing Scientific Community game in action as a board game. • Want to play the game in class. SCG(HSR)

32. Highest Safe Rung • You are doing stress-testing on various models of glass jars to determine the height from which they can be dropped and still not break. The setup for this experiment, on a particular type of jar, is as follows. SCG(HSR)

33. Highest Safe Rung Bob Alice You have a ladder with n rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung. You have a fixed ``budget'' of k > 0 jars. Only two identical bottles to determine highest safe rung (k=2) SCG(HSR)

34. Highest Safe Rung Bob Alice HSR(9,2)≤4 I doubt it: refutation attempt! Alice constructs decision tree T of depth 4 and gives it to Bob. He checks whether T is valid. Bob wins if he finds a flaw. Only two identical bottles to determine highest safe rung SCG(HSR)

35. SCG Scenario • Interactions between scholars Alice and Bob. Admin Nina gives grade to performance of Alice and Bob. SCG(HSR)

36. HSR(n,k)≤q • There exists a valid decision tree DT-HSR(n,k) of depth q to solve HSR(n,k) so that for all ladders with n rungs and for all secret rungs s, the decision tree DT-HSR(n,k) correctly identifies s. SCG(HSR)

37. x Linear Search: HSR(4,1)=3 no yes y z 1 u highest safe rung 0 2 1 3 3 2 depth is 3 SCG(HSR)

38. x no Binary Search: HSR(4,2)=2 yes y z 2 u highest safe rung 1 3 0 3 1 2 SCG(HSR)

39. Pos. HSR Use Case: HSR(n,k) <= q • Name: HSR • Participating actors: Alice, Bob and Nina. • Entry condition: n,k,q are given; k<=n, q<=n, refuter defined: Bob. • Flow of events SCG(HSR)

40. Pos. HSR Use Case (continued) • Flow of events • Alice claims HSR(n,k)<=q. • Bob tries to refute. Bob asks for program/algorithm for (n,k) (ProvideProblem). • Alice provides program/algorithm (SolveProblem). • Bob/Nina check correctness of program/algorithm. • Nina gives grade based on whether program/algorithm is correct and of predicted quality. SCG(HSR)

41. Pos. HSR Use Case (continued) • Exit condition: winner and loser are determined. • Quality requirements: programming language, computational model: decision tree SCG(HSR)

42. Neg. HSR Use Case: HSR(n,k) > q • Name: HSR-neg • Participating actors: Alice, Bob and Nina. • Entry condition: n,k,q are given; k<=n, q<=n, refuter defined: Bob. • Flow of events SCG(HSR)

43. Neg. HSR Use Case (continued) • Flow of events • Alice claims HSR(n,k)>q. • Bob tries to refute. Alice asks for program/algorithm for (n,k) (ProvideProblem). • Bob provides program/algorithm (SolveProblem). • Alice/Nina check correctness of program/algorithm. If depth of decision tree is <= q, refutation is successful. • Nina gives grade based on whether program/algorithm is correct and of predicted quality. SCG(HSR)

44. Neg. HSR Use Case (continued) • Exit condition: winner and loser are determined. • Quality requirements: programming language, computational model: decision tree SCG(HSR)

45. HSR(x,1)<=x-1 x no yes y z 1 u highest safe rung 0 2 1 3 2 x-1 depth is x-1 x-2 x-1 SCG(HSR)

46. Bob has the following claims • HSR(4,1)<=4 • HSR(9,2)<=4 • HSR(9,2)<=3 • HSR(8,3)<=3 • HSR(4,2)<=2 • HSR(11,2)<=4 • HSR(12,2)<=4 Alice makes a decision for each claim: defendable/refutable (refute function) defendable: Alice provides decision tree and Bob cannot find a bug. refutable: Bob provides decision tree and Alice finds a bug. To make the game more interesting: defendable claims are treated first If defendable, can it be strengthened? SCG(HSR)

47. Play Game in class(abbreviated rules) • Role Alice (1-3 students from class) • Role Bob (the rest of class) • Role Nina (3 students from class) • Alice chooses two claims: HSR(9,2)<=3, HSR(11,2)<=4 that she thinks she can refute. • Now play! Intro SCG

48. Who is the winner? • Nina keeps score. • Initially Alice and Bob have 10 points. Intro SCG

49. Bob has the following claims • HSR(4,1)<=4 • HSR(9,2)<=4 • HSR(9,2)<=3 • HSR(8,3)<=3 • HSR(4,2)<=2 • HSR(11,2)<=4 • HSR(12,2)<=4 Alice makes a decision for each claim: defendable/refutable (refute function) defendable: Alice provides decision tree and Bob cannot find a bug. refutable: Bob provides decision tree and Alice finds a bug. To make the game more interesting: defendable claims are treated first SCG(HSR)

50. Focus on • HSR(11,2)<=4 • Alice provides decision tree. • HSR(12,2)<=4 SCG(HSR)