Coevolution

1. Coevolution Competition and Cooperation Padmini Rajagopalan Aditya Rawal

2. OUTLINE • What is Coevolution? • Evolution and Coevolution • Competitive Coevolution • Cooperative Coevolution • Ideas!!

3. What is Coevolution?Inspiration from Nature Fig 1: Coevolution in Heliconius and Passion Flower “An evolutionary change in a trait of individuals of one population in response to a trait of individuals of a second population, followed by an evolutionary response of the second population to a change in the first.” - H. E. Evans, 1984

4. Coevolution in Nature - Egg mimicry in Passiflora Heliconius lays eggs on Passiflora leaves Passiflora leaves evolved to be toxic Heliconius larvae evolve to become resistant to toxin Passiflora evolved to produce yellow spots on its leaves resembling butterfly eggs Fig 3: Fake eggs produced by the plant, on vine Fig 4: Coevolutionary mechanism Fig 2: Heliconius eggs on vine

5. What is Coevolution?Computer Science Perspectives • Definition: two different populations evolve simultaneously and their fitness is measured based on their interactions with the other population [4] • Sorting Network Diagram [3] 2 Independent Species Sorting Network Testcase Fitness Function Evolve Evolve Species interact through fitness function Fig 5: Coevolution – Application in Sorting Networks

6. Evolution and Coevolution Fitness Fitness Fitness Population 2 Population 1 Population Fitness Evaluation Fitness Evaluation Fitness Evaluation Selection Selection Problem Domain Model Problem Domain Model Crossover Crossover Mutation Mutation Next Generation Next Generation Fig 6: Evolution and Coevolution

7. Coevolution Through Competition • Arms Race • Host fitness depends on how many parasites it defeats and vice-versa • Host and Parasite have different objectives • Remember the Hamburger game? (From Risto’s slides) Host Population Parasite Population Fitness Fitness Each Individual can be a Neural Network or chromosome Population 2 Fitness Evaluation Fitness Evaluation Competition between individuals being evaluated Evolve Evolve Fig 7: Arms Race in Competitive Coevolution

8. Problems in Sustaining an Arms Race • Loss of gradients [4] • Over-specialization • Red queen dynamics or relativism • Intransitivity

9. Competitive Coevolution Improvements • Spatial Coevolution [4] [6] [7] • The hosts and parasites are distributed on a toroidal grid and each of them interacts only with the hosts/parasites that are located close to it on the grid. Fig 8: Spatial Coevolution Performance

10. Competitive Coevolution Improvements • Hall of fame - The best parasites from each generation are saved and hosts in each new generation are tested against a random sample of these. • Resource sharing, also known as competitive fitness sharing [2] [9] [8] • Host is considered "fitter" if it defeats a parasite that few other hosts have defeated. • Initialization with "good" genes - The host population can be initialized with some good features.

11. Advantages of Competitive Coevolution • It doesn't get stuck at local optima as often as evolution. • Since the environment is constantly evolving, the solutions don't stagnate. • Has higher efficacy. That is, it seems to discover high-level strategies [5] • Coevolution requires sparse training rather than relying on large datasets like evolution • Helps preserve diversity over longer periods [6]

12. Competitive Coevolution – Applications • This is related to work on sorting networks [3] • self-playing Backgammon learner [Tesauro, 1992] • coevolving life-forms [Sims, 1994], • co-evolving Tic-tac-toe players [Angeline & Pollack, 1993] • Financial Market simulations

13. Coevolution through Cooperation Arch 1 Enforced Subpopulations (ESP) • Each neuron in the Hidden Layer is represented by a subpopulation that evolves independently • Fitness of the Network is passed to all the neurons equally • Each neuron population converges to a specific role so as to maximize the fitness of the Neural Network [12] This Neural Network can be an individual in the Host/Parasite example Problem Domain Model Fig 9: Enforced Neuron Subpopulations [12]

14. Coevolution through Cooperation Arch 2 Multi-agent ESP [12] • The Neural Network from the previous slide can represent a Predator in Predator-Prey domain • Predators cannot catch the Prey greedily without a strategy • Fitness is equally distributed amongst the Predators • Each predator adapts to specific role • Notice the Hierarchical Layers of Cooperation Fig 10: Multi-agent Enforced Subpopulations [12]

15. Coevolution through Cooperation Arch 2 Multi-agent ESP [12] • Autonomous Communicating agents vs. Autonomous Non-Communicating agents • Role based interaction better than Direct communication between agents • Strategy for Non-communicating team – 2 agents acts as chasers and 1 agent acts as a blocker (Video) Fig 11: Average number of generations required to solve the task [12]

16. Coevolution through Cooperation Arch 3Evolving Coadapted Subcomponents [11] • Decomposition of problem into sub-tasks followed by optimization • Can Coevolve various modalities of a behavior within a single robot or behavior of a group of cooperating robots Fig 12: Evolving Coadapted Subcomponents [11]

17. Emergence of Cooperation Using Evolution [10] • Goal is to achieve cooperation among multiple agents of colony, during a foraging task • Higher rewards assigned to the cooperative task • Can extend this architecture to coevolution – by coevolving colonies Each individual can be a Neural Network Colony 1 Colony 2 Fitness Fitness Fitness Evaluation Fitness Evaluation Problem Domain Crossover Problem Domain Next Generation of Colony 2 Next Generation of Colony 1 Fig 13: Evolutionary Architectures to achieve Cooperation

18. Emergence of Cooperation using Evolution[10] • Individual Level Selection vs. Colony Level Selection • Homogeneous Colony vs. Heterogeneous Colony Fig 14: Evolutionary Architectures to achieve Cooperation [10] • Altruism, Division of Labor emerges (video) • Can extend these architectures to coevolution – by coevolving colonies

19. Applications of Cooperative Coevolution • Better performance in team objectives [12] • Simulate colony behavior – herding, hunting, signaling, teaching and learning/following • Team games – like Unreal Tournament • Development of Altruistic individuals [10]

20. Ideas • What if prey evolves? • Using both competition and cooperation as both means and ends • Emergence of Cooperation in multi-agent systems • Prey capture domain which is a special case of pursuit-evasion domain • Team Dynamics - Self-interest vs. Team interests • Self allocation of tasks in Multi-agent systems

21. References [1] Grefenstette, J. and Daley, R. (1995); Methods for Competitive and Cooperative Evolution, ICMAS'95 (pp. 276-282), AAAI Press. [2] Rosin, C. D. & Belew, R. K. (1995); Methods for competitive coevolution: Finding opponents worth beating. In Eshelman, L. J. (Ed.), Proceedings of the Sixth International Conference on Genetic Algorithms, pp. 373-381. San Mateo, CA: Morgan Kaufmann. [3] Hillis, D. W. (1990); Co-evolving parasites improve simulated evolution as an optimization procedure. Physica D 42:228-234. [4] Mitchell, M., Thomure, M. D., and Williams, N. L. (2006); The role of space in the success of coevolutionary learning. In In L. M. Rocha et al. (editors), Artificial Life X: Proceedings of the Tenth International Conference on the Simulation and Synthesis of Living Systems, pp. 118--124. Cambridge, MA: MIT Press. [5] Pagie, L. & Mitchell, M. (2002); A comparison of evolutionary and coevolutionary search. International Journal of Computational Intelligence and Applications, 2(1), 53--69.

22. References continued [6] Mitchell, M. (2006); Coevolutionary learning with spatially distributed populations. In G. Y. Yen and D. B. Fogel (editors), Computational Intelligence: Principles and Practice. New York: IEEE Computational Intelligence Society. [7] Williams, N. and Mitchell, M. (2005); Investigating the success of spatial coevolutionary learning. In H. G. Beyer et al. (editors), Proceedings of the 2005 Genetic and Evolutionary Computation Conference, GECCO-2005, New York: ACM Press, 523-530. [8] Juillé, H. and Pollack, J. B. (1998); Coevolutionary learning: A case study. Proceedings of the Fifteenth International Conference on Machine Learning, Madison, Wisconsin, July 24 - 26, 1998, pp 251-259. [9] C. Rosin and R. Belew (1996); New methods for competitive coevolution. Evolutionary Computation, vol. 5, no. 1. [10] Perez-Uribe, A., Floreano, D. and Keller, L. (2003); Effects of group composition and level of selection in the evolution of cooperation in artificial ants. 7th European Conference on Artificial Life (ECAL'2003), pp. 128-137.

23. References continued [11] Potter, M. A., and Jong, K. A. D. (2000); Cooperative coevolution: An architecture for evolving coadapted subcomponents. Evolutionary Computation, 8:1–29. [12] C. Yong and R. Miikkulainen (2007); Coevolution of role-based cooperation in multi-agent systems. Technical Report AI07-338, The University of Texas at Austin Department of Computer Sciences.

24. Demos • http://nn.cs.utexas.edu/demos/multiagent-esp/rolebased.gif • http://lis.epfl.ch/research/projects/EvoAnts/videos/EvoShort-lowres.mpg