Human-Level AI for Computer Games

1. Human-Level AI for Computer Games John E. Laird University of Michigan laird@umich.edu http://ai.eecs.umich.edu/people/laird

2. Why I’m Interested in Video Games • AI is going to be more important in games • A great environment for developing human-level AI • Academic AI has something to offer the game industry

3. Applications of AI to Games • Synthetic Characters • Sports games, FPS, RTS, … • Dynamic Game Control • Modify play calling based on experience (coaching) • Directing characters for plot and story • Providing play-by-play, camera control, … • Development of Games • Automatic testing

4. Human-Level AI & Games • FPS Enemies: Quake, Unreal, Half-Life, ... • FPS Partners: Spec Ops, Rainbow Six • Adventure Characters: Blade Runner • RPG Characters: Baldur’s Gate • Massively Multiplayer Games: Asheron’s Call, Everquest

5. Human Behavior ModelingTime Scales Seconds 104 1 hour Game 103 10 min Level 102 1 min Engagement 101 10 sec Unit task 100 1 sec Operations 10-1 100 msec Deliberative Act 10-2 10 msec Neural Circuit 10-3 1 msec Neuron 10-4 100 msec Organelle Rational Band Cognitive Band Neural Band

6. Human Behavior As A Sequence Of Deliberative Actions • Consensus: • Human task-level behavior (> 100 milliseconds) can be model as a sequence of deliberate actions, each taking approximately 50 milliseconds. • Basic Perception takes > 100 milliseconds. • Simplest action (key press) takes 70 milliseconds. • “Press a key when the light goes on.” Perception Thinking Action Total 100 msec. + 50 msec. + 70 msec. = 220 msec. We found reaction time (shooting or sidestepping) to enemy in Quake II to be .5-2.5 seconds

7. Processing is Pipelined • Perception/Cognition/Motor can run in parallel Perception Cognition Motor

8. Dual Task: Tracking/Choice

12. Cognitive Architectures • ACT-R (John Anderson, Carnegie Mellon) • Detailed models of behavior and learning • 10 milliseconds-10 seconds. • Used by cognitive psychologists • Issues on performance and scalabiltiy • EPIC (David Kieras, University of Michigan) • Detailed models of human-computer interaction (no learning) • Emphasizes models of perception and action • 50 milliseconds-10 seconds • Used by cognitive psychologist • Soar (John Laird, University of Michigan) • Models of human behavior and learning • 50 milliseconds-1 hour • Emphasis more on complex behavior and learning • Used by AI researchers and cognitive psychologist

13. Conflicting Goals for Human-level AI in Games Goal Driven Reactive Human Characteristics Knowledge Intensive Low CPU & Memory Usage Fast & Easy Development

14. Soar/Games Project • Build an AI Engine around the Soar AI architecture • Soar/Quake II • Soar/Descent 3 • Soar/Half-life • Build autonomous, unscripted AIs Quake II Sensor Data Interface DLL AI Engine(Soar) KnowledgeFiles Actions

15. Soar • AI engine to support multi-method problem solving • Applied to wide variety of tasks and methods • Proposed unified theory for modeling human cognition • Models a wide variety of human behavior: language, HCI, ... • Combines reactive and goal-directed symbolic processing • Supports very large bodies of knowledge (>100,000 rules) • Optimized implementation in ANSI C • In the public domain

16. Soar Quakebot • Goal: Human-like behavior • Fun and challenging to play against • Not super-human • Currently plays as well as good player • Completely implemented in Soar • > 750 rules • Spread across operators for exploring, wandering, chasing, … • Sensing similar to human • Runs on separate PC from game • Uses ~5-10% of 400 MHz Win-95/98/NT machine

17. Simulated Perception Self: orientation, position, health Senses: vision, hearing, smell, taste, touch Augmented Senses: radar, night-vision, … > 80 sensors in Quakebot Field of Vision Obstacle Simulated Actions Movement, weapons’ use, communication ~20 actions in Quakebot Execution Flow of an AI Engine Sense G A M E Think Act

18. Execution Flow of an AI Engine Sense Finite-state machines Subsumption Neural nets Rule-based systems C code Planning systems G A M E ? Think Act

19. Rule-based Systems • Program = set of if-then rules, not sequential code. • Use whatever knowledge that is relevant to current situation Rule Memory Tactics, Movement Knowledge, ... [ ] [ ][ ] [ ] [ ] [ ] [ ][ ] [ ] [ ] [ ] [ ] [ ][ ] Match Changes Working Memory sensing, elaborations, persistent memories, goals, actions …

20. Rules • Conditions contain variables and other first-order tests • If sense a weapon and do not have that weapon, pickup the weapon. (p rule1 (sense weapon ?weapon1) (self weapon <> ?weapon1)  (action pickup ?weapon1)) • A single rule can have multiple instantiations • A single condition can match multiple working memory elements • Compilers convert rules into discrimination networks • Only test changes to working memory. • Share tests across similar rules • Efficient matching

21. Which Rule Should Fire? • Matching finds which rules can fire in current situation. • Conflict resolution picks which rules actually fire. • Based only on syntactic features of rules and data. • Must engineer rules so correct rule is selected each time. Rule Instantiations Rule Memory Conflict resolution [ ] [ ] ? Match [ ] [ ] Working Memory [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]

22. Weaknesses in Rules • Conflict resolution based on syntactic features • Not knowledge of the task • Rules combine three types of knowledge about actions • What is possible • What should be done (based on conflict resolution) • How to do it Leads to duplication if have multiplicity in one of the types • No hierarchical organization of knowledge and goals

23. Soar’s Approach • Organize knowledge as operators: • Primitive & abstract actions • Turn, move, climb, goto-door, get-item, wander, attack, chase • Provides trace of behavior comparable to human decisions • Use rules to propose, select, and apply operators • Rules fire in parallel - act as an associative memory • Provides fine-grain, situation-dependent behavior • Conditional selection and execution determined dynamically Elaboration (propose operators) Decide (select operator) Elaboration (apply operator) Elaboration (terminate operator & propose) Decide Decide Output Output Input Input Input

24. Execution Flow of Soar Engine Sense Soar Elaboration G A M E Propose Operators Think Evaluate Proposed Operators Select One Operator Perform Operator Actions Act

25. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

26. Soar Quakebot Top Operators Attack Retreat Wander Explore Record Enemy Remove Enemy Chase

27. Soar Quakebot Top Operators Attack Retreat Wander Explore Record Enemy Remove Enemy Chase If enemy visible and my health is > very-low-health-value (20%) and his weapon is not much better than mine THEN propose attack If enemy visible and my health is < very-low-health-value (20%) or his weapon is much better than mine THEN propose retreat If no enemy visible or recorded and explored level THEN propose wander If no enemy visible or recorded and not explored level THEN propose explore

28. Soar Quakebot Top Operators Attack Retreat Wander Explore Record Enemy Remove Enemy Chase If enemy just became not visible and no enemy visible or recorded THEN propose record-enemy If hear enemy and no enemy visible or recorded THEN propose record-enemy If record-enemy selected THEN record enemy’s last position and time + 20 seconds. If time > recorded enemy’s saved time THEN propose remove-enemy If remove-enemy is selected THEN remove recorded enemy structure If there is a recorded enemy structure THEN propose chase

29. Example Rules Attack Get-item Face-enemy Side-step Approach Shoot IF I see a weapon that is much better than any I have THEN I need that weapon IF my health is less than low-health-value (40%) THEN I need any health object IF attacking an enemy and there is an object I need and that object is closer than attack-get-item-range THEN propose get-item for that object IF two get-item operators are proposed THEN prefer selecting the one for the closer object

30. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

31. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

32. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

33. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

34. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

35. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

36. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

37. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

38. Get item Go through door Get item in room Exit room Go to door Face item Stop moving to item Move to item Detect item missing Face door Align with door Record at door Move to door Shoot Slide to door Stop move to door Stop slide to door

39. Example Tactics • Collect-Powerups • Pick up best weapons from spawn locations • Remember when missing items will respawn • Use shortest paths to get objects • Get health/armor if low on health/armor • Pickup up other good weapons/ammo if close by to deny enemy • Attack • Use circle-strafe • Move to best distance for current weapon • Chase enemy based on sound of running • Ambush in corner that can’t be seen by enemy • Hunt at nearest spawn room after killing enemy

40. Mapping • Need information on location of walls, doors, etc. • Many tactics require this information • Built up through exploration of a level • Similar to a robot exploring with range sensors • Saved and reused when return to level • Represented internally as a graph structure • Currently restricted to 2D rectangles

43. Anticipation? Dennis (Thresh) Fong: “Say my opponent walks into a room. I'm visualizing him walking in, picking up the weapon. On his way out, I'm waiting at the doorway and I fire a rocket two seconds before he even rounds the corner. A lot of people rely strictly on aim, but everybody has their bad aim days. So even if I'm having a bad day, I can still pull out a win. That's why I've never lost a tournament.” Newsweek, 11/21/99

44. Anticipation • Predicts enemy’s actions • Creates internal model of enemy’s state • Uses own knowledge to predict what enemy would do • Can be modulated by knowledge of enemy’s tactics • Stops planning when it knows what to do or uncertainty in actions • Uses prediction to select actions: • Ambush, Deny-Powerups

45. ?

47. His Distance: 1 My Distance: 1

48. His Distance: 2 My Distance: 2

49. His Distance: 2 My Distance: 2

50. His Distance: 3 My Distance: 1 (but hall)