Evolution, Brains and Multiple Objectives

1. Evolution, Brains and Multiple Objectives By Jacob Schrum schrum2@cs.utexas.edu

2. About Me • B.S. from S.U. in 2006 • Majors: Math, Computer Science and German • Honors Thesis w/ Walt Potter: Genetic Algorithms and Neural Networks • Currently Ph.D. student at U.T. Austin • Received M.S.C.S. in 2009 • Neural Networks Research Group: Genetic Algorithms and Neural Networks

3. Evolution • Change in allele frequencies in population • Alleles = variant gene forms • Genes ⇨ traits • Traits affect: • Survival • Reproduction • Natural selection favors good traits

4. Genetic Algorithms • Abstraction of evolution • Genes = bits, integers, reals • Natural selection = fitness function • Mutation = bit flip, integer swap, random perturbation, … • Crossover = parents swap substrings • Other representations, mutation ops, crossover ops, …

5. Applications Boolean Satisfiability K. A. De Jong and W. M. Spears, “Using Genetic Algorithms to Solve NP-Complete Problems” ICGA 1989

6. Applications Magic Squares T. Xie and L. Kang, "An evolutionary algorithm for magic squares" CEC 2003

7. Applications Circuit Design J. D. Lohn and S. P. Colombano, "A circuit representation technique for automated circuit design" EC 3:3 Sep. 1999

8. Applications Wing Design/Cost Optimization J. L. Rogers and J. A. Samareh, "Cost Optimization with a Genetic Algorithm" NASA Langley Research Center, RTA 705-03-11-03, October 2000

9. Applications Traveling Salesman Problem P. Jog, J. Y. Suh, and D. van Gucht. "The effects of population size, heuristic crossover and local improvement on a genetic algorithm for the traveling salesman problem" ICGA 1989.

10. Applications Resource-Constrained Scheduling S. Hartmann, "A competitive genetic algorithm for resource-constrained project scheduling" NRL 45 1998

11. Applications Lens Design X. Chen and K. Yamamoto, "Genetic algorithm and its application in lens design", SPIE 1996

12. Applications Weight Selection for Fixed Neural Networks F.H.F. Leung, H.K. Lam, S.H. Ling and P.K.S. Tam, "Tuning of the structure and parameters of a neural network using an improved genetic algorithm" NN 14:1 Jan. 2003

13. Applications What Are Neural Networks?

14. Artificial Neural Networks • Brain = network of neurons • ANN = simple model of brain • Neurons organized into layers

15. What Can Neural Networks Do? • In theory, anything! • Universal Approximation Theorem • NNs are function approximators • In practice, learning is hard • Supervised: Backpropagation • Unsupervised: Self-organizing maps • Reinforcement Learning: Temporal-difference learning and Evolutionary computation

16. Neuro-Evolution • Genetic Algorithms + Neural Networks • Many different network representations Fixed length string Subpopulations for each Evolve topology and weights hidden layer neuron [1] [2] [1] F. Gomez and R. Miikkulainen, "Incremental Evolution Of Complex General Behavior" Adaptive Behavior 5, 1997. [2] K. O. Stanley and R. Miikkulainen, "Evolving Neural Networks Through Augmenting Topologies" EC 10:2, 2002.

17. Constructive Neuroevolution • Population of networks w/ no hidden nodes • Random weights and connections

18. Constructive Neuroevolution • Evaluate, assign fitness • Select the fittest to survive

19. Constructive Neuroevolution • Fill out population • Crossover and/or cloning Crossover Clone

20. Constructive Neuroevolution • Random mutations • Perturb weight, add link, splice neuron No mutation Perturb weight Add link Splice neuron

21. Constructive Neuroevolution • Can add recurrent links as well • Provide a form of memory

22. Neuroevolution Applications Double Pole Balancing F. Gomex and R. Miikkulainen, “2-D Pole Balancing With Recurrent Evolutionary Networks” ICANN 1998

23. Neuroevolution Applications Robot Duel K. O. Stanley and R. Miikkulainen, "Competitive Coevolution through Evolutionary Complexification" JAIR 21, 2004

24. Neuroevolution Applications Vehicle Crash Warning System N. Kohl, K. Stanley, R. Miikkulainen, M. Samples, and R. Sherony, "Evolving a Real-World Vehicle Warning System" GECCO 2006

25. Neuroevolution Applications http://nerogame.org/ Training Video Game Agents K. O. Stanley, B. D. Bryant, I. Karpov, R. Miikkulainen, "Real-Time Evolution of Neural Networks in the NERO Video Game" AAAI 2006

26. What I Do With Neuroevolution • Discover complex behavior • Multiagent domains • Simulations, robotics, video games • Support for multiple modes of behavior • Multiobjective optimization

27. Mutiobjective Optimization • Pareto dominance: iff • Assumes maximization • Want nondominated points • NSGA-II [3] used • Popular EMO method Nondominated [3] K. Deb, S. Agrawal, A. Pratap, T. Meyarivan, "A Fast Elitist Non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II" PPSN VI, 2000

28. Non-dominated Sorting Genetic Algorithm II • Population P with size N; Evaluate P • Use mutation to get P´ size N; Evaluate P´ • Calculate non-dominated fronts of {P È P´} size 2N • New population size N from highest fronts of {P È P´}

29. Evolve Game AI • Game where opponents have multiple objectives • Inflict damage as a group • Avoid damage individually • Stay alive individually • Objectives are contradictory and distinct Opponents take damage from bat Player is knocked back by NPC

30. Intelligent Baiting Behavior

31. How to avoid stagnation • Some trade-offs are too easy to reach • Focus on difficult objectives • TUG: Targeting Unachieved Goals • Avoids need for incremental evolution Evolution Hard Objectives

32. Smaller Team w/ Expert Timing

33. Multitask Domains • Perform separate tasks • Predator/Prey • Prey: run away • Pred: prevent escape • Front/Back Ramming • Attack with ram on front • Attack with ram on back

34. Multimodal Networks • One network, multiple policies • Multitask [4] = one mode per task • Mode mutation = network chooses mode to use Multitask Mode Mutation Two tasks, two modes Start with one mode, mutation adds another Appropriate mode used for task Preference neurons control mode choice [4] R. A. Caruana, "Multitask learning: A knowledge-based source of inductive bias" ICML 1993

35. Multimodal Predator/Prey Behavior Learned with Mode Mutation Runs away in Prey task Corralling behavior in Predator task

36. Multimodal Front/Back Ramming Behavior Learned with Multitask Efficient front ramming Immediately turn around to attack with back ram

37. What about “real” domains? • Unreal Tournament 2004 • Commercial video game • Basis for BotPrize competition: Bot Turing Test • Placed 2nd with our bot: UT^2

38. UT^2 Behavior/Judging Game

39. Summary • Neural networks can represent complex behavior • Neuroevolution = way to discover this behavior • Multiobjective evolution needed in complex domains • Success in challenging designed/commercial domains

40. Questions?E-mail: schrum2@cs.utexas.edu Webpage: http://www.cs.utexas.edu/~schrum2/

41. Auxiliary Slides • Empirical results

42. Differences for Alternating and Chasing significant with p < .05