Human Competitive Results of Evolutionary Computation

1. Human Competitive Results of Evolutionary Computation Presenter: Mati Bot Course: Advance Seminar in Algorithms (Prof. YefimDinitz)

2. Human Competitive Results of Evolutionary ComputationOutline • Human-Competitiveness Definition • Evolving Hyper-Heuristics using Genetic Programming • Rush-Hour (bronze Humies prize in 2009 by Ami Hauptman) • Freecell(gold Humies prize in 2011 by AchiyaElyasaf) • Other Examples

3. What is Human competitive? • John Koza defined “Human-Competitiveness” in his book: Genetic Algorithms IV (2003). • There are 8 criteria by which a result can be considered Human-Competitive.(will be explained in next slide) • Our mission: Creation of Human-Competitive innovative solutions by means of Evolution.

4. The 8 Criteria of Koza for Human Competitivenes. • (A) result is a Patent from the past, improvement of a patent. A New patent.

5. The 8 Criteria of Koza for Human Competitivenes. • (B) result is equal to or better than another result that was published in a journal.

6. The 8 Criteria of Koza for Human Competitivenes. • (C) result is equal to or better than a result in a known DB of results.

7. The 8 Criteria of Koza for Human Competitivenes. • (D)publishable in its own right as a new scientific result.  • independent of the fact that the result was mechanically created.

8. The 8 Criteria of Koza for Human Competitivenes. • (E) The result is equal to or better than the best human-created solution.

9. The 8 Criteria of Koza for Human Competitivenes. • (F)equal to or better than an human achievement in its field at the time it was first discovered.

10. The 8 Criteria of Koza for Human Competitivenes. • (G) The result solves a problem of indisputable difficulty in its field.

11. The 8 Criteria of Koza for Human Competitivenes. • (H) The result holds its own or wins a regulated competition involving human contestants (in the form of either live human players or human-written computer programs).

12. Humies Competition • “Humies annual Competition” gives $$$ for the best HC results. (in GECCO conference) • Awarding a gold, silver and bronze prizes to the best entries. (money $$$) • BGU won 1 gold, 1 silver and 6 bronze prizes since 2005. • I counted more than 75 Human-Competitive results on the Humies competition site. • http://www.sigevo.org/gecco-2012/ • http://www.genetic-programming.org/hc2011/combined.html

13. Evolving Hyper-Heuristics using Genetic Programming Ami Hauptman and AchiyaElyasaf

14. Overview • Introduction • Searching Games State-Graphs • Uninformed Search • Heuristics • Informed Search • Evolving Heuristics • Test Cases • Rush Hour • FreeCell

15. Representing Games as State-Graphs • Every puzzle/game can be represented as a state graph: • In puzzles, board games etc., every piece move can be counted as an edge/transition between states • In computer war games etc. – the place of the player / the enemy, all the parameters (health, shield…) define a state

16. Rush-Hour as a state-graph Move purple Move blue

17. Searching Games State-GraphsUninformed/naïve Search • BFS – Breadth First Search • Optimal solution • Exponential space in the search depth • DFS– Depth First Search(without node coloring). • We might “never” track down the right path. • Usually games contain cycles • Linear Space • Iterative Deepening: Combination of BFS & DFS • Each iteration DFS with a depth limit is performed. • Limit grows from one iteration to another • Worst case - traverse the entire graph

18. Iterative Deepening

19. Searching Games State-GraphsUninformed Search • Most of the game domains are PSPACE-Complete! • Worst case - traverse the entire graph • We need an informed-search! (or an intelligent approach to traversing the graph)

20. Searching Games State-GraphsHeuristics • Heuristic function h:states -> Real. • For every state s, h(s) is an estimation of the minimal distance/cost from s to a solution • In case h is perfect: • an informed search that tries states with the lowest h-value first – will simply stroll to a solution • For hard problems, finding a good h is hard • Bad heuristic means the search might never track down the solution • We need a good heuristic function to guide the informed search

21. Searching Games State-Graphs Informed Search • Best-First search: Like DFS but select nodes with higher heuristic value first • Not necessarily optimal

22. 1 2 3 4 Best-First Search

23. Searching Games State-Graphs Informed Search • A*: • G(s)=cost from root till s • H(s)=Heuristic estimation • F(s)=G(s)+H(s) • Holds closed and sorted open lists(the list of states needs to be checked out). • Best (=lowest F(s)) node of all open nodes is selected.

24. A* 1 2 3 4

25. Overview • Introduction • Searching Games State-Graphs • Uninformed Search • Heuristics • Informed Search • Evolving Heuristics • Previous Work • Rush Hour • FreeCell

26. Evolving Heuristics • For H1, … ,Hn – heuristics building blocks.How should we choose the fittest heuristic? • Minimum? Maximum? Linear combination? • GA/GP may be used for: • Building new heuristics from existing building blocks • Finding weights for each heuristic (for applying linear combination) • Finding conditions for applying each heuristic

27. Evolving Heuristics: GA • Genotype(The coding of a solution in a Population) – • Phenotype(The meaning of the coding) –

28. Evolving Heuristics: GP If False Condition True * And + H5 / H2 * ≤ ≥ H1 0.1 H1 0.1 H1 0.4 H2 0.7

29. Evolving Heuristics: Policies

30. Evolving Heuristics: Fitness Function

31. Overview • Introduction • Searching Games State-Graphs • Uninformed Search • Heuristics • Informed Search • Evolving Heuristics • Test cases • Rush Hour • FreeCell

32. Rush Hour GP-Rush [Hauptman et al, 2009] Bronze Humies award

33. Domain-Specific Heuristics • Hand-Crafted Heuristics / Guides: • Blocker estimation – lower bound (admissible) • Goal distance – Manhattan distance • Hybrid blockers distance – combine the above two • Is Move To Secluded – did the car enter a secluded area? (last move blocks all other cars) • Is a Releasing Move – if the last move increased the number of free cars.

34. Blockers Estimation • Lower bound for number of steps to goal • By: Counting moves to free blocking cars Example: • O is blocking RED • Need at least: • Move O • Move C • Move B • Move A  H = 4

35. Goal Distance 16 Deduce goal Use “Manhattan Distance” from goal as h measure

36. Hybrid 16+8=24 “Manhattan Distance” + Blockers Estimation

37. Policy “Ingredients” Functions & Terminals:

38. Results Average reduction of nodes required to solve test problems, with respect to the number of nodes scanned by iterative deepening: H1: the heuristic function BlockersLowerBound. H2: GoalDistance. H3: Hybrid. Hc is our hand-crafted policy. GP is the best evolved policy, selected according to performance on the training set.

39. Results (cont’d) Time (in seconds) required to solve problems JAM01 . . . JAM40: ID – iterative deepening, Hi – average of our three hand-crafted heuristics, Hc – our hand-crafted policy. GP – our best evolved policy. human players (average of top 5).

40. FreeCell FreeCell remained relatively obscure until Windows 95 There are 32,000 solvable problems (known as Microsoft 32K), except for game #11982, whichhas been proven to be unsolvable Evolving hyper heuristic-based solvers for Rush-Hour and FreeCell [Hauptman et al, SOCS 2010] GA-FreeCell: Evolving Solvers for the Game of FreeCell [Elyasaf et al, GECCO 2011]

41. FreeCell (cont’d) • As opposed to Rush Hour, blind search failed miserably • The best published solver to date solves 96% of Microsoft 32K • Reasons: • High branching factor • Hard to generate a good heuristic

42. Learning Methods: Random Deals Which deals ((חלוקות קלפיםshould we use for training? First method tested - random deals • This is what we did in Rush Hour • Here it yielded poor results • Very hard domain

43. Learning Methods: Gradual Difficulty hard easy Second method tested - gradual difficulty • Sort the problems by difficulty • Each generation tests solvers against 5 deals from the current difficulty level + 1 random deal

44. A few words on Co-evolution Population 1 Population 2 Test for fitness Test for fitness Examples? • Problems, adversaries, Examples: • Freecell Deals • Rush Hour Boards • Another Chess Player • Solution, Solvers. Examples: • Freecell Solver • Rush Hour Solver • Chess player

45. Learning Methods: Hillis-Style Co-evolution Third method tested - Hillis-style co-evolution using “Hall-of-Fame”: • A deal population is composed of 40 deals (=40 individuals) + 10 deals that represent a hall-of-fame • Each hyper-heuristic is tested against 4 deal individuals and 2 hall-of-fame deals

46. Learning Methods: Rosin-style Co-evolution p1 p2 p1 Fourth method tested - Rosin-style co-evolution: • Each deal individual consists of 6 deals • Mutation and crossover: • Crossover: • Mutation

47. Results

48. Other Human Competitive results • Antenna Design for the International Space Station • Automatically finding patches using genetic programming • Evolvable Malware • And many more on Humies site.

49. Thank you for listening any questions?