NU ACM Talk Virtual Scientific Communities for Driving Innovation and Learning

1. Supported by Novartis and GMO NU ACM Talk Virtual Scientific Communities for Driving Innovation and Learning Karl Lieberherr joint work with Ahmed Abdelmeged and Bryan Chadwick SCG Innovation

2. Introduction • Scientific Community Game(X) [SCG(X)] • Goal: Foster innovation and learning in some domain X • A virtual scientific community consists of virtual scholars that propose and oppose hypotheses maximizing their reputations • Applications: Learning and innovation through focused interaction, “Netflix in the small” Innovation

3. Your Programming Assignments • Input f in F satisfying precondition • Output g in G satisfying postcondition • Claims about your program p: F -> G: • Correctness • Resource consumption • Time: <= |f|*log(|f|) for |f|>100. • Space: <= 10 • Energy • Quality of G: 0.6

4. Programming Assignments • Further claims about your program p: F -> G: • Well designed • Follows well accepted design guidelines

5. Evaluating Programming Assignments • Run them on a benchmark provided by the teaching assistant / professor. • Run them on all test cases provided by other students. Reward students whose test cases find bugs in programs of other students. Good idea! Increases interaction. • What about checking resource consumption and quality claims? Require that they are stated explicitly as part of the submission.

6. Hypotheses • I claim that my program p:F->G is correct for all inputs satisfying the precondition and it operates with resource consumption R and achieves quality Q. • Example hypothesis: I claim that my sorting program is correct for all inputs of size n < 10^6 and it makes at most n*log(log(n)) comparisons.

7. How to model a scholar? • Solve problems • Provide hard problems • Propose hypotheses about Solve and Provide (Introspection) • Oppose hypotheses • Strengthen hypotheses • Refute hypotheses • Supported opposing failed • Refuted opposing succeeded Innovation

8. Where SCG comes from • J ACM • SCG = Scientific Community Game • SCG = Specker Challenge Game Lieberherr/Specker 1981 SCG Innovation

9. Outline • Introduction (done) • Highest safe rung example • SCG Scholar / Agent • SCG Agent in Action • Highest safe rung example (opposition) • Who is the winner? • Competition and collaboration • Disadvantages of SCG • Further Examples • SCG-based Software Development Process • Conclusions SCG Innovation

10. Example: Jar Stress Testing • You have a ladder with r rungs, and you want to find the highest rung from which you can drop a copy of the jar and not have it break.  We call this the highest safe rung problem (r,b). • How many experiments do you need? Minimize. • (r,infinity) • (r,1) SCG

11. Highest Safe Rung ProblemProblems and Solutions • Problems: p=((r,b),secrethsr), secret hsr in [0,r], r,b natural numbers • r = number of rungs • b = number of jars that are allowed to break • (r,b) is called a niche • Solutions: sequence of queries of the form n? to find hsr. Responses: yes/no. • Quality of solution: q = length of sequence of queries SCG Innovation

12. Highest Safe Rung ProblemHypotheses (from Kleinberg/Tardos) • Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality q: abbreviated H = ((r,b),q) • Problems to be delivered for H = ((r,b),q) are of the form ((r,b), s). Important: A hypothesis defines a family of problems. • Propose: Hypotheses H1 = ((25,2),11), H2 = ((25,2),6) SCG Innovation

13. Life of a scholar: (propose+ oppose+ provide* solve*)* Scholars propose and oppose proposed hypotheses egoistic Alice egoistic Bob social welfare HB1 opposes (1) HA1 HB2 provides problem (2) HA2 solves problem LOSES WINS! not as well as she expected based on HA2 (3) HA3 Bob increases his reputation HA4 Innovation

14. What is the purpose of SCG? • The purpose of playing an SCG(X) competition is to assess the "skills" of the agents in: • solving problems in domain X, • making good predictions about niches in domain X, • finding the hardest problems in a specific niche of X Innovation

15. What is SCG(X) agent Alice agent Bob degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation agent plays transfer to reliable, efficient software more applications: test constructive knowledge Innovation

16. What is SCG(X)? Team Alice Team Bob Teams Design Problem Solver Develop Software Deliver Agent I am the best No!! Agent Alice Agent Bob Let’s play constructively Administrator SCG police Innovation

17. For agents: Full Round Robin Tournaments or Swiss-Style • Agents to play the SCG(X). Repeat a few times with feedback used to update agents. • Within the group of participating agent, the winning agent has the • best solver for X-problems • best supported knowledge about X Innovation

18. SCG in Action: Competitions • http://www.ccs.neu.edu/home/lieber/courses/cs4500/f09/files/competitions/past_competitions/11_23/tournament_1/final_results_tournament_2009_11_24_12_03_41.html • http://www.ccs.neu.edu/home/lieber/courses/cs4500/f09/files/competitions/past_competitions/10_22/tournament_1/final_results_tournament_2009_10_23_04_35_18.html Innovation

19. Highest Safe Rung Problemopposing • opposing(refuting, strengthening) • Alice claims: Hypothesis ((25,2),5) • Bob opposes it by refuting it: Bob invents problem ((25,2), secret 9). Alice: 5? no, 10? yes, 6? no, 7? no, 8? no. Already 5 questions asked and answer still unknown. Alice’ claim is refuted. • Alice claims: Hypothesis ((25,2),12) • Bob opposes it by strengthening it to ((25,2),9); and he can successfully support this hypothesis SCG Innovation

20. Highest Safe Rung Problemsupporting • Alice claims: Hypothesis ((25,2),12) • Bob tries to oppose but Alice supports it: Bob provides: ((25,2), secret 23). Alice: 5? no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes. Only 9 questions asked and problem ((25,2), secret 23) is solved. Alice has supported her hypothesis. SCG Innovation

21. Motivated by real scientific community Who wins? Alice or Bob? • Reputation of Alice = • the number of hypotheses that Alice proposed that were never successfully opposed by Bob (neither refuted nor strengthened) + • the number of hypotheses that Bob proposed that were successfully opposed by Alice • RA = HAnotOpposedB + HBOpposedA • The scholar with the highest reputation wins • encourages: creating strong knowledge and opposing knowledge created by others SCG Innovation

22. competitive / collaborative Agent Alice: claims hypothesis H Alice wins knowledge Agent Bob: opposes H, refutes: provides evidence for !H Bob wins reputation makes public knowledge Innovation

23. Highest Safe Rung Problemcompetition / collaboration • Alice claims: Hypothesis ((25,2),12) • Bob tries to oppose but Alice supports it: Alice: 5? no, 10? no, 15? no, 20? no, 25? yes, 21? no, 22? no, 23? no, 24? yes. • From this exchange which is prompted by Alice defending her reputation, Bob gets an idea: For problem: p=((r,b),secret hsr), consider f(r,x) =(r/x + x) and find a x so that f(25,x) is minimized. f(25,5)=10; f(25,6)=11;f(25,4)=11. • From this idea Bob knows that he can strengthen the hypothesis to ((25,2),10) • General solution for 2 jars to break: Given r, find x to minimize (r/x + x). SCG Innovation

24. Scholars and Agents:Same rules • Are encouraged to • offer results that are not easily improved. • offer results that they can successfully support. • strengthen results, if possible. • expose results that are wrong. • stay active and publish new results. • be well-rounded: solve posed problems and pose difficult problems for others. • become famous! Innovation

25. Soundness Theorem • SCG is sound: The agent with the best algorithms / knowledge wins (there is no way to cheat) • best: within the group of participating agents Innovation

26. Highest Safe Rung ProblemAsymptotic Hypotheses • Alice claims the hypothesis: I can solve any problem p=((r,b),secret hsr) with quality f(r,b) : abbreviated H = ((r,b),f(r,b)) • Problems to be delivered for H = ((r,b),f(r,b)) are of the form ((r,b), secret hsr). • Propose: Hypotheses H1 = ((r,b),(log(r))b), H2 = ((r,b),r1/b) SCG Innovation

27. Highest Safe Rung Problemopposing asymptotic hypothesis • opposing (refuting, strengthening) • Alice claims: Hypothesis ((r,b),(b*log(r))) • Bob opposes it by refuting it: Bob invents problem ((1024,2), secret hsr). log(1024) = 10. 20 questions are not enough! Alice: 30? no, 60? yes, 31? no, 32? no, etc.. Already 20 questions asked and answer still unknown. Alice’ claim is refuted. • Alice claims: Hypothesis ((r,2),r/2) • Bob opposes it by strengthening it to ((r,2),2*r½ ); and he can successfully support this hypothesis. SCG Innovation

28. Disadvantages of SCG • The game is addictive. After Bob having spent 4 hours to fix his agent and still losing against Alice, Bob really wants to know why! • Overhead to learn to define and participate in competitions. • The administrator for SCG(X) must perfectly supervise the game. Includes checking the legality of X-problems. • if admin does not, cheap play • watching over the admin Innovation

29. How to compensatefor those disadvantages • Warn the scholars. • Use a gentleman’s security policy: report administrator problems, don’t exploit them to win. • Occasionally have a non-counting “attack the administrator” competition to find vulnerabilities in administrator. • both generic as well as X-specific vulnerabilities. Innovation

30. GIGO: Garbage in / Garbage out • If all agents are weak, no useful solver created. Innovation

31. Physics Maximum Height ProblemProblems and Solutions • Problems: p=(v, a), v, a: positive real numbers • The maximum height obtained by a projectile launched with speed v at angle a to the horizontal is z. • Solutions: real number z. • Quality of solution: Number of correct decimal places. SCG Innovation

32. Physics Maximum Height Problem Hypotheses • Alice claims the hypothesis: I can solve any maximum height problem p=(v,a) with quality q in 1 minute: abbreviated H = (MHP,q) • Problems to be delivered for H = (MHP,q) are of the form (v,a). • Propose: Hypotheses H1 = (MHP,3), H2 = (MHP,6) http://scienceworld.wolfram.com/physics/Height.html SCG Innovation

33. Physics Maximum Height Problem opposing • opposing (refuting, strengthening) • Alice claims: Hypothesis (MHP,3) • Bob opposes it by refuting it: Bob invents problem (25,60 degrees). Alice fails to solve the problem in 1 minute with 3 correct digits. Alice’ claim is refuted. Checking is done by experiment or trusted third party. • Alice claims: Hypothesis (MHP,1) • Bob opposes it by strengthening it to (MHP,2); and he can successfully support this hypothesis SCG Innovation

34. RegExpToAutomata ProblemProblems and Solutions • Problems: p=(r,n); r a regular expression of size n. • r = regular expression; a + b* a + a a a b* • n defines a niche of regular expressions • Solutions: DFA d equivalent to r. • Quality of solution: Number of states of d. SCG Innovation

35. RegExpToAutomata ProblemProblems and Solutions • Problems: p=(r,n); r a regular expression of size n. • r = regular expression; a + b* a + a a a b* • n defines a niche of regular expressions • Solutions: DFA d equivalent to r. • Quality of solution: Number of states of d. SCG Innovation

36. RegExpToAutomata Problem Hypotheses • Alice claims the hypothesis: I can solve any problem p=(r,n) with quality q or less: abbreviated H = (n,q) • Problems to be delivered for H = (n,q) are of the form p=(r,n). Important: A hypothesis defines a family of problems. • Propose: Hypotheses H1 = (5,11), H2 = (5,10) SCG Innovation

37. RegExpToAutomata Problem opposing • opposing(refuting, strengthening) • Alice claims: Hypothesis (5,11) • Bob opposes it by refuting it: Bob invents a regular expression r of size 5, gives it to Alice and she fails to deliver a DFA d with 11 or fewer states. Alice’ claim is refuted. • Alice claims: Hypothesis (5,20) • Bob opposes it by strengthening it to (5,19); and he can successfully support this hypothesis SCG Innovation

38. RegExpToAutomata Problem supporting • Alice claims: Hypothesis (4,12) • Bob tries to oppose but Alice supports it: Bob gives to Alice a regular expression r of size 4. Alice provides and equivalent DFA with 12 or fewer states. Alice has supported her hypothesis. SCG Innovation

39. Who wins? Alice or Bob? • Reputation of Alice = • the number of hypotheses that Alice proposed that were never successfully opposed by Bob (neither refuted nor strengthened) + • the number of hypotheses that Bob proposed that were successfully opposed by Alice. • RA = HAnotOpposedB + HBOpposedA • The scholar with the highest reputation wins. • encourages: creating minimum automata for regular expressions of a given size. SCG Innovation

40. Software Development Process • Increase targeted interaction between software developers. Innovation

41. Traditional Approach Requirements for X Human Developers human1 human3 human2 human4 Static Evaluation. No competition. Testing unit testing integration testing Develop new software for problem solving domain X Benchmark is used to evaluate software Users SCG-SP2010

42. Why Software Development through a virtual scientific community? Requirements for X Evaluates fairly, frequently, constructively and dynamically. Drives innovation. Challenges humans. Agents point humans to what needs attention in the software. Human Developers human1 human2 Erika Patrick SCG(X) Erika-Patrick-agent winning-agent Develop new software for problem solving domain X Benchmark is used to evaluate software Users SCG-SP2010

43. Erika-Patrick Agent • Surrogate of combined knowledge of Erika and Patrick successfully transferred to agent. • Transfer knowledge by programming. SCG-SP2010

44. Conclusions • How to make learning and problem solving fun: design a game and interact. • Scientific Community Game = Specker Challenge Game = SCG • How to create reliable problem solving software? Have it tested through SCG. Innovation

45. Final Slide • More Questions? Innovation

46. SCG Innovation

47. SCG concepts • Scholars working in a domain with niches. Define functions on niches. • Hypotheses: claims about functions on niches: • Discounting protocol for HA: Alice selects niche element ne and Bob applies fBob so that claim about function does not hold • Strengthening protocol • Reputation SCG Innovation

48. SCG concepts • Scholars working in a domain with niches. Function f: Niche -> S for Alice and Bob. • Hypotheses: claims about niches: belief: f has property b(s, dn, fdn). (Niche,Belief) • Discounting protocol: Alice selects niche element ne and Bob applies fBob creating s, so that !b(s,ne) • Strengthening protocol • Reputation SCG Innovation

49. Hypothesis Structure • Algorithm Solver: Problems -> Solutions • For all p in Problems with feature f in Features algorithm Solver solves p using resources p(f) with quality(p,Solver(p),f). • Algorithm Provider: Features -> Problems • For feature f, Algorithm Provider provides a problem p, for all solutions of p, !quality(p,Solver(p),f). SCG Innovation

50. Two person SCG • Alice, Bob • Domain: Source, Target; fA, fB-> Source-> Target; Source defined by niche predicate. • Hypotheses HA (HB): claims about fA (fB) • Discounting protocol for HA: • Bob provides element ne in Source so that fA(ne) contradicts HA. • Alice provides element ne in Source so that fB(ne) contradicts HA. • Strengthening protocol • Bob proposes HB, HA => HB and Alice cannot discount HB. SCG Innovation