1 / 27

CAP6938 Neuroevolution and Developmental Encoding Competitive Coevolution

Explore the use of neuroevolution and developmental encoding in competitive coevolution to solve complex problems. Learn how to choose opponents, evaluate performance, and promote innovation and complexity in evolving agents.

doherty
Download Presentation

CAP6938 Neuroevolution and Developmental Encoding Competitive Coevolution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CAP6938Neuroevolution and Developmental EncodingCompetitive Coevolution Dr. Kenneth Stanley October 16, 2006

  2. Example:I Want to Evolve a Go Player • Go is one of the hardest games for computers • I am terrible at it • There are no good Go programs either (hypothetically) • I have no idea how to measure the fitness of a Go player • How can I make evolution solve this problem?

  3. Generally: Fitness May Be Difficult to Formalize • Optimal policy in competitive domains unknown • Only winner and loser can be easily determined • What can be done?

  4. Competitive Coevolution • Coevolution: No absolute fitness function • Fitness depends on direct comparisons with other evolving agents • Hope to discover solutions beyond the ability of fitness to describe • Competition should lead to an escalating arms race

  5. The Arms Race

  6. The Arms Race is an AI Dream • Computer plays itself and becomes champion • No need for human knowledge whatsoever • In practice, progress eventually stagnates (Darwen 1996; Floreano and Nolfi 1997; Rosin and Belew 1997)

  7. So Who Plays Against Whom? • If evaluation is expensive, everyone can’t play everyone • Even if they could, a lot of candidates might be very poor • If not everyone, who then is chosen as competition for each candidate? • Need some kind of intelligent sampling

  8. Challenges with Choosing the Right Opponents • Red Queen Effect: Running in Circles • B dominates A • C dominates B • A dominates C • Overspecialization • Optimizing a single skill to the neglect of all others • Likely to happen without diverse opponents in sample • Several other failure dynamics

  9. Heuristic in NEAT:Utilize Species Champions Each individual plays all the species champions and keeps a score

  10. Hall of Fame (HOF)(Rosin and Belew 1997) • Keep around a list of past champions • Add them to the mix of opponents • If HOF gets too big, sample from it

  11. More Recently:Pareto Coevolution • Separate learners and tests • The tests are rewarded for distinguishing learners from each other • The learners are ranked in Pareto layers • Each test is an objective • If X wins against a superset of tests that Y wins against, then X Pareto-dominates Y • The first layer is a nondominated front • Think of tests as objectives in a multiobjective optimization problem • Potentially costly: All learners play all tests De Jong, E.D. and J.B. Pollack (2004). Ideal Evaluation from Coevolution Evolutionary Computation, Vol. 12, Issue 2, pp. 159-192, published by The MIT Press.

  12. Choosing Opponents Isn’t Everything • How can new solutions be continually created that maintain existing capabilities? • Mutations that lead to innovations could simultaneously lead to losses • What kind of process ensures elaboration over alteration?

  13. Alteration vs. Elaboration

  14. Answer: Complexification • Fixed-length genomes limit progress • Dominant strategies that utilize the entire genome must alter and thereby sacrifice prior functionality • If new genes can be added, dominant strategies can be elaborated, maintaining existing capabilities

  15. Test Domain: Robot Duel • Robot with higher energy wins by colliding with opponent • Moving costs energy • Collecting food replenishes energy • Complex task: When to forage/save energy, avoid/pursue?

  16. Robot Neural Networks

  17. Experimental Setup • 13 complexifying runs, 15 fixed-topology runs • 500 generations per run • 2-population coevolution with hall of fame (Rosin & Belew 1997)

  18. Performance is Difficult to Evaluate in Coevolution • How can you tell if things are improving when everything is relative? • Number of wins is relative to each generation • No absolute measure is available • No benchmark is comprehensive

  19. Expensive Method: Master Tournament(Cliff and Miller 1995; Floreano and Nolfi 1997) • Compare all generation champions to each other • Requires n^2 evaluations • An accurate evaluation may involve e.g. 288 games • Defeating more champions does not establish superiority

  20. Strict and Efficient Performance Measure: Dominance Tournament (Stanley & Miikkulainen 2002)

  21. Result: Evolution of Complexity • As dominance increases so does complexity on average • Networks with strictly superior strategies are more complex

  22. Comparing Performance

  23. Summary of Performance Comparisons

  24. The Superchamp

  25. Cooperative Coevolution • Groups attempt to work with each other instead of against each other • But sometimes it’s not clear what’s cooperation and what’s competition • Maybe competitive/cooperative is not the best distinction? • Newer idea: Compositional vs. test-based

  26. Summary • Picking best opponents • Maintaining and elaborating on strategies • Measuring performance • Different types of coevolution • Advanced papers on coevolution: Ideal Evaluation from Coevolution by De Jong, E.D. and J.B. Pollack (2004)Monotonic Solution Concepts in Coevolution by Ficici, Sevan G. (2005)

  27. Next Topic: Real-time NEAT (rtNEAT) • Simultaneous and asynchronous evaluation • Non-generational • Useful in video games and simulations • NERO: Video game with rtNEAT -Shorter symposium paper: Evolving Neural Network Agents in the NERO Video Game by Kenneth O. Stanley and Risto Miikkulainen (2005)-Optional journal (longer, more detailed) paper: Real-time Neuroevolution in the NERO Video Game by Kenneth O. Stanley and Risto Miikkulainen (2005) -http://Nerogame.org -Extra coevolution papers

More Related