CAP6938 Neuroevolution and Developmental Encoding Evolving Adaptive Neural Networks

1. CAP6938Neuroevolution and Developmental EncodingEvolving Adaptive Neural Networks Dr. Kenneth Stanley October 23, 2006

2. Remember This Thing? What’s missing from current neural models?

3. An ANN Link is a Synapse (from Dr. George Johnson at http://www.txtwriter.com/Backgrounders/Drugaddiction/drugs1.html )

4. What Happens at Synapses? • Weighted signal transmission • But also: • Strengthening • Weakening • Sensitization • Habituation • Hebbian learning • None of these weight changes during a lifetime are happening in static models

5. Why Should Weights Change? • The world changes • Evolution cannot predict all future possibilities • Evolution can succeed with less accuracy • The Baldwin Effect • Learning smooths the fitness landscape • Traits that initially require learning eventually become instinct if the environment is consistent • If the mind is static, you can’t learn!

6. How Should Weights Change? • Remember Hebbian Learning? (lecture 3) • Weight update based on correlation: • Incremental version: • How can this be made to evolve? • Which weights should be adaptive? • Which rule should they follow if there is more than one? • Which weights should be fixed? • To what degree should they adapt (evolve alpha) • Evolve alpha parameter on each link

7. Floreano’s Weight Update Equations • Plain Hebb Rule: • Postsynaptic rule: • Weakens synapse if postsynaptic node fires alone • Presynaptic rule: • Covariance rule: Strengthens when correlated, weakens when not

8. Floreano’s Genetic Encoding

9. Experiment: Light-switching • Task: Go to black area to turn on light, then go to area under light • Requires a policy change in mid-task: Reconfigure weights for new policy Fully Recurrent Network Blynel, J. and Floreano, D. (2002) Levels of Dynamics and Adaptive Behavior in Evolutionary Neural Controllers. In B. Hallam, D. Floreano, J. Hallam, G. Hayes, and J.-A. Meyer, editors. From Animals to Animats 7: Proceedings of the Seventh International Conference on Simulation on Adaptive Behavior, MIT Press.

10. Results • Adaptive synapse networks evolved straighter and faster trajectories • Rapid and appropriate weight modifications occur at the moment if change

11. However, It’s Not That Simple • A recurrent network with fixed synapses can change its policy too • The activation levels cycling through the network are a kind of memory that can affect its functioning • Do we need synaptic adaptation at all? • Experiment in paper: Kenneth O. Stanley, Bobby D. Bryant, and Risto Miikkulainen (2003). Evolving Adaptive Neural Networks with and without Adaptive Synapses, Proceedings of the 2003 IEEE Congress on Evolutionary Computation (CEC-2003).

12. Experimental Domain: Dangerous Food Foraging • Food may be poisonous or may not • No way to tell at birth • Only way to tell is to try one • Then policy should depend on “pain” or not

13. Condensed Floreano Rules • Two adaptation rules: One for excitatory connections, the other for inhibitory: • First term is Hebbian, second term is a decay term

14. NEAT Trick: Use “Traits” to Prevent Dimensionality Multiplication • One set of rules/traits • Each connection gene points to one of the rules • Rules evolve in parallel with network • Weights evolve as usual

15. Robot NNs

16. Surprising Result • Fixed-weight recurrent networks could evolve a solution more efficiently! • Adaptive networks found solutions, but more slowly and less reliably

17. Explanation • Fixed networks evolved a “trick”: Strong inhibitory recurrent connection on left turn output causes it to stay on until it experiences pain. Then it turns off and robot spins (from right turn output) until it doesn’t see food anymore, and it runs to the wall • In adaptive network, 22% of connections diverge after pain, causing network to spin in place: a holistic change

18. Discussion • Adaptive neurons are not for everything, not even all adaptive tasks • In non-adaptive tasks, they only add unnecessary dimensions to the search space • In adaptive tasks, they may be best for tasks requiring holistic solutions • What are those? • Don’t underestimate the power of recurrence

19. Next Topic: Leaky Integrator Neurons, CTRNNs, and Pattern Generators • Real neurons encode information as spikes and spike trains with differing rates • Dendrite may integrate spike train at different rates • Rate differences can create central pattern generators without a clock! Levels of dynamics and adaptive behavior in evolutionary neural controllers by Blynel, J., and Floreano, D. (2002) Evolution of Central Pattern Generators for Bipedal Walking in a Real-Time Physics Environment by Torsten Reil and Phil Husbands (2002) Optional: Evolution and analysis of model CPGs for walking I. Dynamical modules by Chiel, H.J., Beer, R.D. and Gallagher, J.C. (1999)