Real-time neuroevolution in the NERO video game

1. Real-time neuroevolution in the NERO video game

2. overview • Background on video games • Neural networks • NE • NEAT • rtNEAT • NERO

3. background • The global video game market, according to analysts, will increase from $25.4 billion in revenue in 2004 to almost $55 billion in 2009 , larger than even that of Hollywood. • Techniques in artificial intelligence can potentially both increase the longevity of video games and decrease their production costs.

4. background • In most games character behavior is scripted. no matter how many times the player exploits a weakness, that weakness is never repaired. • So what to do?

5. The solution • Machine learning can keep video games interesting by allowing agents to change and adapt. • The agents can be trained to perform complex behaviors. • if they would be trained offline (out-game learning) and then “freeze” and put in the game, they will not adapt and change in response to players in the game.

6. The solution • That’s why agents should adapt and change in real-time, and a powerful and reliable machine learning method is needed. • Prior examples in the MLG genre include the Tamagotchi virtual pet • and the video “God game”Black & White.

7. The solution • Here I introduce the machine learning : rtNEAT – real time Neuroevolution Of Augmenting Topologies. • And the game that implement the rtNEAT: NERO – neuroEvolving robotic operatives.

8. Weighted weights Neural Networks • A try to somehow mimic the neural network of a human being in a simplistic way. • The network is a collection of simple units that mimic a neuron: input weights Weighted weights Computing unit output 2 threshold 0.2 Σ 1.5

9. Neural Networks • The network adapts as follows: change the weight by an amount proportional to the difference between the desired output and the actual output: • where η is the learning rate, D is the desired output, and Y is the actual output

10. NE – Neuroevolution • The artificial evolution of neural networks using an evolutionary algorithm. • The network can compute arbitrarily complex functions, can both learn and perform in the presence of noisy inputs, and generalize its behavior to previously unseen inputs.

11. Demands of video games AI • Large state/action space: high dimensional space and having to check the value of every possible action on every game tick for every agent in the game. - In NE agents are evolved to output only a single requested action per game tick

12. Demands of video games AI 2)Diverse behaviors: Agents should not all converge to the same Behavior - A homogeneous population would make the game boring. - In NE Diverse populations can be explicitly maintained through speciation

13. Demands of video games AI 3)Consistent individual behaviors: Characters should take a random action in order to explore new behaviors, but not periodically making odd moves. - In NE The behavior of an agent does not change because it always chooses actions from the same network.

14. Demands of video games AI 4)Fast adaptation and sophisticated behaviors: not waiting hours for agents to adapt require simple presentation but that results in simple behaviors. - In NE A representation of the solution can be evolved, allowing simple behaviors in the beginning and complexified later.

15. Demands of video games AI 5)Memory of past states: in order to react more convincingly to the present situation requires keeping track of more than the current state. - In NE Recurrent neural networks can be evolved, what gives us a memory of past situations.

16. NEAT • A technique for evolving neural networks using an evolutionary algorithm. • The correct topology not needed to be known prior to evolution. • Unique in that it begins with minimal networks and adds connections and nodes to them over generations, allowing complex problems to be solved based upon simple ones.

17. Genetic encoding - Genome

18. Genetic encoding - Mutation

19. Genetic encoding - Mutation

20. Genetic encoding - results • The splitting inserting into the system a nonlinearity. • Old behaviors that were in the pre existing networks are not vanished and their quality stay almost the same, with the opportunity to refine those behaviors.

21. Cross over – historical markings • When a mutation occurs a global innovation number is incremented and assigned. • When cross over happens the offspring will inherit the innovation numbers. • Thus the historical origin of every gene is known throughout the evolution.

22. Crossover – historical markings • With innovation numbers we don’t need to do a topological analysis. • Two genes with the same historical origin represent the same structure since they were both derived from the same ancestral gene at some point in the past

23. Crossover – historical markings

24. Speciation • NEAT divides the population into species. • The individuals compete primarily with their own niches. • this way topological innovations are protected and have time to optimize their structure before competing with other niches, like new ideas that reach a potential before elimination.

25. Speciation • Another advantage that Species with smaller genomes survive as long as their fitness is competitive, ensuring that small networks are not replaced by larger ones unnecessarily.

26. Speciation

27. Speciation • To examine the similarity of networks a distance is formulated: • E = number of excess, D = number of disjoint, W = average weight differences of matching genes, N = num of genes in the larger Genome (a normalizing factor) • c1,c2 and c3 defines the importance of the three factors.

28. Speciation • if a genome’s distance to a randomly chosen member of the species is less than δt, a compatibility threshold, the genome is placed into this species. • δt can be dynamic, be raised if there are too many species and lowered if too few. • If a genome is not compatible with any existing species, a new species is created.

29. Speciation – adjusted fitness • The fitness of an organism is also defined by its niche: • Where • And you sum over all the distances from the organisms in the population resulting in the number of organisms in the same species as organism i.

30. Speciation - offspring • The number of offspring distributed to each species k is: • Where Fk is the average fitness of species k,Ftot = Σk Fk is the total of all species fitness average, and |P| is the size of the population

31. Speciation - offspring • First we eliminate the lowest performing members of the population. • The whole generation is then replaced by the new offspring.

32. NEAT – minimization and complexification • NEAT begins with uniform simple networks with no hidden nodes, differing only by their initial random weights. • It adds connections and nodes incrementally. • Only the most fitted structures remains. • NEAT searches for the optimal topology by complexifying existing network.

33. NEAT - performance • In experiments it has been shown that all three main components of NEAT are essential for it to work – the historical markings, speciation, and starting from minimal structure. • It was also shown that NEAT outperform other NE methods, especially those that are fixed topology evolution because it find faster complex solutions by starting with simple networks and expanding only when beneficial.

34. rtNEAT • In order for players to interact with evolving agents in real time NEAT was improved to rtNEAT. • Now fitness statistics are collected constantly as the game is played, and the agents evolved continuously as well.

35. rtNEAT – replacement cycle • Every few game ticks the worst individual is replaced by an offspring of parents chosen from among the best.

36. rtNEAT - algorithm • Because rtNEAT performs in real time it cannot produce the whole generation at once like NEAT, and so the algorithm loop must change.

37. rtNEAT - algorithm

38. rtNEAT - algorithm • Calculating adjusted fitness: • Where |S| is the number of individuals in the species.

39. rtNEAT - algorithm • Removing the worst Agent:a) if we remove the worst unadjusted fitness agent, innovation preservation will be damaged, because new small species will be eliminated as soon as they appear.That’s why the worst adjusted fitness agent will be removed.

40. rtNEAT - algorithm • Removing the worst Agent:a) if we remove the worst unadjusted fitness agent, innovation preservation will be damaged, because new small species will be eliminated as soon as they appear.That’s why the worst adjusted fitness agent will be removed.

41. rtNEAT - algorithm • Removing the worst Agent:b) agents must also have time to evaluated sufficiently before removed, because unlike NEAT that each individual live the same amount of time, in rtNEAT different agents have been around for different amount of time.That’s why rtNEAT only removes agents that have played for more than the minimum amount of time m.

42. rtNEAT - algorithm • Removing the worst Agent:c) we must re-estimate the average fitness F, because a specie now have one less member, and it’s average most likely been changed.

43. rtNEAT - algorithm • Creating offspring:the probability to choose a parent specie is proportional to it’s average fitness compared with the total of all species’ average fitnesses:And a single new offspring is created by recombining two individuals from the parent species

44. rtNEAT - algorithm • Reassigning agents to species:because minimizing the number of species is important in real time environment, when CPU time is not allocated all to evolution, the threshold δt must be dynamically adjusted.But also the entire population must be assigned again to species.

45. rtNEAT - algorithm • Replacing the old agent with the new one:replacing depends on the game.You may just replace the neural network in the body of the removed agent. Or you may kill it and replace a new agent instead.

46. rtNEAT – loop interval • The loop interval should be every n ticks. For n to be chosen a “law of eligibility” is formulated: • Where m is the minimum time alive, n is the number of ticks between replacement (the loop interval), |p| is the population size, and I is the fraction of the population that is too young to be evaluated.

47. rtNEAT – loop interval • And I left for the user to determine, because its most critical for performance.

48. NERO Neuroevolving Robotics Operatives • the idea of the game is to train a team of agents by designing a training program. • In order for the agents to learn, the learning algorithm is of course rtNEAT.

49. NERO - fitness • In the training there are sliders that lets you configure what attributes you want your agents to have.

50. NERO - fitness • fitness is the sum of all the components ci multiply by the sliders value vi: • To put in also the rate of forgetting r into the fitness this formula is given: • Where ftis the current fitness.