Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA

1. The Scientific Community Game: Education and Innovation Through Survival in a Virtual World of Claims Supported by Novartis • Karl Lieberherr • Northeastern University • College of Computer and Information Science • Boston, MA • joint work with • Ahmed Abdelmeged and Bryan Chadwick

2. Why Scientific Community Game(SCG) • … motives in academic publishing: • desire for recognition and respect from the people one regards as peers, • desire to have impact (on conclusions being reached, on the development of the discipline, etc.), and • desire to participate in significant knowledge-buildingdiscourse. • e.g., Scardamalia, M., & Bereiter, C. (1994) Bionetics 2010

3. SCG is Bio-inspired • Virtual world of scholars based on natural selection • propose, oppose (refute and strengthen) claims • maximize reputation, weak scholars are removed. • Turn problem-solving software into virtual organisms that fend for themselves and survive in a virtual world inhabited by virtual organisms created by your peers. Bionetics 2010

4. SCG is a web-based implementation of Karl Popper’s science ideas • One of the greatest philosophers of science of the 20th century. • Falsifiability or refutability is the logical possibility that an assertion could be shown false by a particular observation or physical experiment. • Error elimination (refutation), performs a similar function for science that natural selection performs for biological evolution. from Wikipedia Bionetics 2010

5. Comparison • Karl Popper: Conjectures and Refutations, 1963 • Scientific Community Game: Claims and Refutations, 2007 Bionetics 2010

6. Recognition in SCG • Scholars build their reputation by proposing and opposing claims, by defending their own claims and refuting or strengthening the claims of others. • The higher their reputation, the more recognition. Bionetics 2010

7. Impact in SCG • Second-order environment • what one scholar does in adapting, changes the environment so that others must readapt. • Developing novel techniques to find superior solutions, challenges others to catch up. Bionetics 2010

8. Knowledge-Building Discoursein SCG • Communication or debate. • Refutation protocol defines the structure of the debate and who wins. Claims are defined through a refutation protocol. • Knowledge-building: • claims that have been defended predominantly are candidates for truth • claims that have been refuted predominantly are probably false. Bionetics 2010

9. Goals of SCG • Put knowledge-building discourse on the web giving participants the option to gain recognition and to have impact. • Focus the discourse through precise definition of claims with refutation protocols. • Make knowledge building discourse fun and educational from the high school to the advanced research level. SCG = Scientific Community Game = Specker Challenge Game Bionetics 2010

10. What do we mean by science? • Science consists of the formulation and testing of hypotheses based on observational evidence. • Ours: Science consists of the formulation and testing of constructive claims based on observational evidence. Construction is computable. Bionetics 2010

11. What do we mean by Scientific Method • Hypothetico-deductive method: Formulate a hypothesis in a form that could conceivably be falsified by a test on observable data. • Ours: Formulate a constructive claim in a form that could conceivably be falsified by a test using a protocol. The refutation protocol is part of the claim to make very explicit when refutation is successful. Bionetics 2010

12. Tartaglia against Fior 1535 Tartaglia was famed for his algebraic solution of cubic equations which was published in Cardan's Ars Magna. Bionetics 2010

13. Outline • Introduction • Popper Science, Renaissance History: Tartaglia and Fior • Definition of SCG • Example (Highest safe rung) • Applications: Teaching, Software Development, Research • Claims with secrets and other protocol variants • Output of SCG, Equilibrium • Advantages and Disadvantages • Conclusions Bionetics 2010

14. Definition of SCG: Domain • Problem: Set • Solution: Set • valid: relation(Problem, Solution) • quality: function(Problem, Solution)->[0..1] Bionetics 2010

15. Claim(Domain) makes predictions about the future • Problems: Powerset(Domain.Problem) • q: Quality = [0,1] • r: Resource = N+ = positive integer Alice claims to have a technique to solve problems in Problems with at least quality q and using at most resources r. Bionetics 2010

16. Implied Protocol of Claim(Domain) • Alice claims (problems,q,r), Bob refutes • Bob provides problem prob in Claim.Problems. • Alice solves problem prob providing sol in Domain.Solution. • check: valid(prob,sol) and quality(prob,sol)>=q and sol.resource<=r. • sol.resource returns Alice’ resource consumption to solve problem prob. Karl Popper: Only hypotheses capable of clashing with observation reports are allowed to count as scientific. Bionetics 2010

17. Claim • Problems: subset of problems • quality in [0,1] 1 quality (how well problems in Problems can be solved) Bionetics 2010 0

18. Claim over strengthening 1 correct valuation quality strengthening Bionetics 2010 18 0

19. Bio-inspired computing: Virtual World of SCG-Avatar • SCG-Avatar (Claim(Domain)) • State: Reputation = positive rational number • Activity • propose new claims • oppose claims of others • refute claim(Problems, q, r) • strengthen claim(Problems, q’, r’), q’>q or r’<r • Reputation gain: refute others’ claims and defend own claims (counter refutation attempts) • Reputation loss: unsuccessful refutation of other’s claim and refutation of own claims Bionetics 2010

20. Tournament • round-robin • Swiss-style • elimination • single • double Bionetics 2010

21. Summary of SCG Definitions Domain Problem Solution valid(Problem, Solution) quality(Problem, Solution) →[0,1] Claim(Domain) Problems: PowerSet(Domain.Problem) q: Quality = [0,1] r: Resource = N+ Rules of the Scientific Community: propose and oppose, be an active scholar, rules for reputation accumulation. Tournaments Bionetics 2010

22. Highest Safe Rung • You are doing stress-testing on various models of glass jars to determine the height from which they can be dropped and still not break. The setup for this experiment, on a particular type of jar, is as follows. Bionetics 2010

23. Highest Safe Rung Bob Alice You have a ladder with n rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break. We call this the highest safe rung. You have a fixed ``budget'' of k > 0 jars. Only two identical bottles to determine highest safe rung Bionetics 2010

24. Highest Safe Rung Bob Alice HSR(9,2)≤4 I doubt it: refutation attempt! Alice constructs decision tree T of depth 4 and gives it to Bob. He checks whether T is valid. Bob wins if he finds a flaw. Only two identical bottles to determine highest safe rung Bionetics 2010

25. x Highest Safe Rung Decision Tree HSR(9,2)=5 no 3 yes y z 1 6 u highest safe rung 0 2 4 9 1 2 3 5 7 9 4 5 8 6 7 8 Bionetics 2010

26. Finding solution for HSR(n,2) Find minimum x, s.t. MaxRungs(x,2) > n • Approximate min x in [0,n] (n/x) + x • Exact • MaxRungs(x,y) =MaxRungs(x-1,y-1)+MaxRungs(x-1,y) • MaxRungs(x, 2) = x + MaxRungs(x – 1, 2) • MaxRungs(0, 2) = 1 • Applied to HSR(9,2) • MaxRungs(3,2) = 7 < 9 • MaxRungs(4,2) = 11 > 9 breaks at root does not break at root Keith Levin CS 4800 Fall 2010 MaxRungs(x,y) = the largest number of rungs we can test with y jars and x experiments. Bionetics 2010

27. MaxRungs • MaxRungs(x,y) = sum [k=0 .. y] binomial(x,k) • All paths are of length x. At most k branches may be left branches. • Note: y = x implies MaxRungs(x,y) = 2x meaning a complete binary tree of depth x. • Example: binomial(3,2)+binomial(3,1)+ binomial(3,0) = 7 Bionetics 2010

28. Formal: HSR • Domain: • Problem: (n,k), k <= n. • Solution: Decision tree to determine highest safe rung. • quality(problem, solution): depth of decision tree / number of rungs • valid(problem, solution): at most k left branches, ... Bionetics 2010

29. Formal: HSR • Claim(Domain): • Alice claims ({(25,2)},9/25,5 seconds) • {(25,2)}: set of problems (singleton) • 9/25: quality • 5 seconds: resource • Refutation Protocol: • Bob refutes: only one problem: (25,2) • Alice: solves problem by providing decision tree t. • predicate: t is a valid decision tree for (25,2) of depth 9 Bionetics 2010

30. Claim involving Experiment Claim ExperimentalTechnique(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. Bionetics 2010

31. Outline • Introduction • Popper Science, Renaissance History: Tartaglia and Fior • Definition of SCG • Example (Highest safe rung) • Applications: Teaching, Software Development, Research • Claims with secrets and other protocol variants • Output of SCG, Equilibrium • Advantages and Disadvantages • Conclusions Bionetics 2010

32. Applications: Software Development • Software Development • Teaching Constructive Domains Bionetics 2010

33. Gamification of Software Development etc. • Want reliable software to solve a computational problem? Design a game where the winning team will create the software you want. • Want to teach a STEM domain? Design a game where the winning students demonstrate superior domain knowledge. Doesn’t TopCoder already do this? STEM = Science, Technology, Engineering, and Mathematics Bionetics 2010

34. SCG and TopCoder • SCG is an abstraction and generalization of what TopCoder does. Bionetics 2010

35. The Traditional Approach Team A Static Benchmark Solver A Ranking Team B Solver B Team C Solver C measure how close to minimum Parameterized by the domain. HSR(9,2)=4 HSR(25,2)=7 Software: Solving HSR Problem: construct decision tree of min. depth Bionetics 2010

36. The Bio-Inspired Approach Dynamic Benchmark Solver A Team A Avatar A Virtual World (Game) prop-opp A Ranking Solver B Team B Avatar B prop-opp B Avatar C Solver C Team C prop-opp C Parameterized by the domain. Bionetics 2010

37. A Virtual WorldAvatar’s View Avatar Claims, Problems, Solutions Opponents’ communication, Feedback • Problems: Benchmark output • Solutions: Software output • Claims: statements about algorithms Administrator Results Bionetics 2010

38. What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? Bionetics 2010

39. SCG = Scientific Community Game • Make software development more scientific. • Software developers build reputation • propose and defend claims about their software • oppose claims made by others • refute claims • strengthen claims • claim includes refutation protocol Bionetics 2010

40. Why a web application with avatars? Fair Evaluation. Who are Alice and Bob? • They are avatars developed by real Alice and real Bob. • Alice and Bob compete with 10 other avatars in a full-round robin tournament. • Who is the winner: The avatar with the highest reputation, i.e., the avatar who has the strongest, not successfully opposed claims (like in a real scientific community). Bionetics 2010

41. our focus What is SCG(X) avatar Bob Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge Bionetics 2010

42. Real Scholars and Avatars:Same rules • Are encouraged to • propose claims that are not easily strengthened. • offer claims that they can successfully support. • strengthen others’ claims, if possible. • stay active and propose new strong claims or oppose others’ claims. • become famous! Bionetics 2010

43. Clear Feedback Sense of Progress What we want Authenticity (Facebook) • Engage software developers • let them produce software that models an organism that fends for itself in a real virtual world while producing the software we want. Have fun. Focus them. • let them propose claims about the software they produce. Reward them when they • defend their claims successfully or • oppose the claims of others successfully. Possibility of Success Bionetics 2010

44. SCG • Gamification of software development for computational problems • A Sociotechnical System for knowledge dissemination, innovation, and integration Bionetics 2010

45. Software Engineering Properties fostered by SCG • Reliable (otherwise the avatar is removed from the game) • Flexible, modular (otherwise the avatar cannot be easily updated between tournaments) • Efficient (otherwise you cannot defend your claims and oppose the claims of others) Adaptive and Aspect-Oriented Software is relevant! Bionetics 2010

46. State of Avatar SCG-Avatar: Our Vision • Companies come to SCG website and define a competition by defining a claim domain X. • Participating teams get baby avatars generated from X that participate in daily competitions. • Competition generates a wealth of information: educated employees, good (undefeated) software, good algorithms, good potential employees. Reward is paid to the winner. Bionetics 2010

47. State of SCG-Avatar: Our Vision • Not only companies but faculty members who want to give their students a rich learning experience for computational problem X. • Or editors of special issues in journals who want to use a competition to get a real world comparison of all approaches to solve computational problem X. Bionetics 2010

48. Life of an avatar: (propose+ oppose+ provide* solve*)* Avatars propose and oppose proposed claims egoistic Alice egoistic Bob social welfare CB1 opposes (1) CA1 CB2 provides problem (2) CA2 solves problem LOSES WINS! not as well as she expected based on CA2 (3) CA3 reputation 10 reputation 1000 transfer 200 CA4 Bionetics 2010

49. What is SCG(X)? Team Alice Team Bob Teams Design Problem Solver Develop Software Deliver Avatar I am the best No!! Avatar Alice Avatar Bob Let’s play constructively Administrator SCG police Bionetics 2010

50. competitive / collaborative Avatar Alice: claim C loses reputation r wins knowledge k Avatar Bob: opposes C, refutes: provides evidence for !C wins reputation r makes public knowledge k Bionetics 2010