Integrating Learning in Interactive Gaming Simulators

1. Integrating Learning in Interactive Gaming Simulators David W. Aha1 & Matthew Molineaux2 1Intelligent Decision Aids Group Navy Center for Applied Research in AI Naval Research Laboratory; Washington, DC 2ITT Industries; AES Division; Alexandria, VA surname@aic.nrl.navy.mil AAAI’04 Workshop on Challenges in Game AI 25 July 2004

2. Outline • Motivation: Learning in cognitive systems • Objectives: • Support empirical studies w/ simulators • Encourage studies that address industry & military M&S concerns • Design: TIELT functionality & components • Example: Knowledge base content • Status: Implementation, collaborations • Summary Thanks to our sponsor:

3. Affect Rough Anatomy of a Cognitive Agent Reflective Processes LTM CognitiveAgent Concepts STM Deliberative Processes Learning Other reasoning Sentences Communication (language, gesture, image) Prediction, planning Perception Action Reactive Processes Sensors Effectors External Environment Attention (Brachman, 2003)

4. Complication The ML research community has been focusing on: ¬Rapid: Knowledge poor algorithms ¬Enduring: Learning over a short time period ¬Embedded: Stand-alone evaluations Status of Cognitive Learning Few deployed cognitive systems integrate techniques that exhibit rapid & enduring learning behavior on complex tasks • It’s costly to integrate & evaluate embedded learning techniques Problem

5. Supervised Learning Supervised Learning ML System ML System Reasoning System Reasoning System Interface (standard format) Interface (standard format) Database Database (e.g., UCI Repository) (e.g., UCI Repository) Cognitive Learning Cognitive Learning Cognitive Learning Reasoning Modules Reasoning Systemk Reasoning Modules ML Module ML Module ML Module Sensors Sensors Sensors ML Module ML Module ML Module World (Simulated/Real) Worldi (Simulated/Real) World (Simulated/Real) Interface (standard API) Interface (standard API) Interface (standard API) ML Module ML Modulej ML Module Effectors Effectors Effectors (e.g., TIELT) (e.g., TIELT) (e.g., TIELT) Testbed for Integrating and Evaluating Learning Techniques Wanted: A New Interface(thanks to W. Cohen, others) Supervised Learning ML Systemj Reasoning Systemk Interface (standard format) Databasei (e.g., UCI Repository of ML Databases)

6. Objectives & Domain Objective Facilitate the evaluation of learning techniques in CogSys • Develop & transition TIELT to help reduce integration costs (time, $) • Support DARPA Challenge Problems on Cognitive Learning • Demonstrate research utility prior to approaching industry/military Domain: Why interactive gaming simulators? • Available implementations (cheap to acquire & run) • Challenging problems for CogSys/ML research • Significant interest (military, industry, academia, funding, public)

7. TIELTSpecification • Simplifies integration & evaluation! • Learning-embedded reasoning systems & gaming simulators • Inputs: 5 descriptions (simulator I/O, game model, learning & performance tasks, reasoning system I/O, & evaluation strategy) • Free • Learning foci: Many • Task (e.g., learn how to execute, or advise on, a task) • Player (e.g., learn/predict a human player’s strategies) • Game (e.g., learn/refine its objects, their relations, & behaviors) • Learning methods: Many types • Supervised/unsupervised, immediate/delayed feedback, analytic, active/passive, online/offline, direct/indirect, automated/interactive • Learning results should be available for inspection • Gaming simulators: Those with challenging learning tasks • Reuse: Provide access to libraries of simulators & reasoning systems • Abstracts interface definitions from game & task models

8. Distinguishing TIELT • Provides an interface for message-passing interfaces • Supports composable system-level interfaces

9. Game Player(s) Reasoning System Library Reasoning System Reasoning System Reasoning System Learning Module Learning Module Learning Module Stratagus . . . . . . . . . Learning Module Learning Module Learning Module Full Spectrum Command TIELT: Integration Architecture TIELT’s User Interface Game Engine Library Advice Interface Evaluation Interface Prediction Interface Coordination Interface TIELT User TIELT’s Internal Communication Modules Selected Game Engine Selected Reasoning System . . . Learned Knowledge (inspectable) TIELT’s KB Editors Selected/Developed Knowledge Bases Game Interface Description Reasoning Interface Description Game Model Description Task Descriptions Evaluation Methodology Description TIELT User Knowledge Base Libraries GMD TDs EMD GID RID GMD TDs EMD GID RID GMD TDs EMD GID RID

10. TIELT’s Knowledge Bases Game Interface Description Defines communication processes with the game engine Reasoning Interface Description Defines communication processes with the learning system Game Model Description • Defines interpretation of the game • e.g., initial state, classes, operators, behaviors (rules) • Behaviors could be used to provide constraints on learning Task Descriptions • Defines the selected learning and performance tasks • Selected from the game model description Evaluation Methodology Description Defines the empirical evaluation to conduct

11. Game Engine Library Game Player(s) Reasoning System Library Reasoning System Reasoning System Reasoning System Processed State Learning Module Learning Module Learning Module Stratagus Raw State Action Decision . . . . . . . . . Learning Module Learning Module Learning Module Full Spectrum Command Example: Controlling a Game Character TIELT’s User Interface Advice Interface Evaluation Interface Prediction Interface Coordination Interface TIELT User TIELT’s Internal Communication Modules Selected Game Engine Selected Reasoning System Learned Knowledge (inspectable) TIELT’s KB Editors Selected/Developed Knowledge Bases Game Interface Description Reasoning Interface Description Game Model Description Task Descriptions Evaluation Methodology Description TIELT User Knowledge Base Libraries GMD TDs EMD GID RID GMD TDs EMD GID RID GMD TDs EMD GID RID

12. Game Engine Library Game Player(s) Reasoning System Library Reasoning System Reasoning System Reasoning System Processed State Learning Module Learning Module Learning Module Stratagus Raw State Edit . . . . . . . . . Learning Module Learning Module Learning Module Full Spectrum Command Edit Example: Updating a Game Model TIELT’s User Interface Advice Interface Evaluation Interface Prediction Interface Coordination Interface TIELT User TIELT’s Internal Communication Modules Selected Game Engine Selected Reasoning System Learned Knowledge (inspectable) TIELT’s KB Editors Selected/Developed Knowledge Bases Game Interface Description Reasoning Interface Description Game Model Description Task Descriptions Evaluation Methodology Description TIELT User Knowledge Base Libraries GMD TDs EMD GID RID GMD TDs EMD GID RID GMD TDs EMD GID RID

13. TIELT’s Internal Communication Modules Database Evaluation Interface Advice Interface Database Engine State Evaluator Controller Stored State Current State Model Updater Learning Translator (Mapper) Translated Model (Subset) Selected Reasoning System Learning Task Selected Game Engine Percepts Action / Control Translator (Mapper) Learning Outputs Actions Perf. Task Game Model Description Game Interface Description Reasoning Interface Description Task Descriptions Evaluation Methodology Description User Game Model Editor Game Interface Editor Reasoning Interface Editor Task Editor Evaluation Editor

14. Sensing the Game State (City placement example) 1 In Game Engine, thegame begins; a colony pod is created and placed. TIELT Current State 2 The Game Engine sends a “See” sensor message identifying the pod’s location. 4 Actions Action Translator 3 Updates Game Engine 1 The Model Updater receives the sensor message and finds the corresponding message template in the Game Interface Description. Sensors 3 5 2 Model Updater Controller 3 4 4 Game Model Description Game Interface Description This message template provides updates (instructions) to the Current State, telling it that there is a pod at the location See describes. User Game Model Editor Game Interface Editor 5 The Model Updater notifies the Controller that the See action event has occurred.

15. Getting Decisions from the Learning System (City placement example) 1 The Controller notifies the Learning Translator that it has received a See message. TIELT Controller 2 The Learning Translator finds a city location task, which is triggered by the See message. It queries the controller for the learning mode, then creates a TestInput message to send to the reasoning system with information on the pod’s location and the map from the Current State. Selected Reasoning System 1 Learning Outputs 4 Action Translator Translated Model (Subset) Learning Module #1 Learning Translator Current State 3 . . . 2 2 Learning Module #n Reasoning Interface Description Task Descriptions 3 The Learning Translator transmits the TestInput message to the Reasoning System. User 4 The Reasoning System transmits output to the Action Translator. Learning Interface Editor Agent Editor

16. Acting in the Game World (City placement example) 1 The Action Translator receives a TestOutput message from the Reasoning System. 4.b, c The Advice Interface receives Move and displays advice to a human player on what to do next, or makes a Prediction. 2 The Action Translator finds the TestOutput message template, determines it is associated with the city location task, and builds a MovePod operator (defined by the Current State) with the parameters of TestOutput. TIELT Advice Interface Prediction Interface 4.b 4.c Current State 1 2 Actions 4.a Action Translator Game Engine 3 3 The Action Translator determines that the MoveAction from the Game Interface Description is triggered by the MovePodOperator and binds Moveusing information from MovePod. 3 2 Game Interface Description Reasoning Interface Description User Game Interface Editor Reasoning Interface Editor 4.a The Game Engine receives Move and updates the game to move the pod toward its destination, or

17. Example Status (July 2004) Initial Work • TIELT v1 + documentation due 9/04 • Message protocols • Current: Console I/O, TCP/IP • Future: Library calls, HLA interface, RMI (possibly) • Message content: Configurable • Instantiated templates tell it how to communicate with other modules • Initialization messages: Start, Stop, Load Scenario, Set Speed • Game Model representations (w/ Lehigh University) • Simple programs • TMK process models • PDDL (language used in planning competitions) • Stratagus/Wargus module (Lehigh University) • Initial publicity (BRIMS’04, here) • Workshops being planned: ICCBR’05 (plus competition), ICML’05, ...?

18. TIELT Collaboration Projects (2004-05)

19. ToEE Platform Library Stratagus U. Minn-D. Lehigh U. Urban Terror FSC/R UT Arl. USC/ICT RoboCup TIELT Collaborations (2004-05) TIELT’s User Interface TIELT User Advice Interface Evaluation Interface Prediction Interface Coordination Interface U.Mich. U.Minn-D. USC/ICT Reasoning System Library Game Library TIELT’s Internal Communication Modules Soar: U.Mich ICARUS: ISLE DCA: UT Arlington EE2 Learning Modules Troika Mad Doc Neuroevolution: UT Austin FreeCiv Others: Many ISLE NWU TIELT’s KB Editors Selected/Developed Knowledge Bases Many LU, USC U. Mich. Mich/ISLE Many Game Interface Description Reasoning Interface Description Game Model Description Task Descriptions Evaluation Methodology Description TIELT User Knowledge Base Libraries

20. Summary: Questions? Concerns? • TIELT: Mediates between a (gaming) simulator and a learning-embedded reasoning system • Goals: • Simplify running learning expts with cognitive systems • Support DARPA challenge problems in learning • Designed to work with many types of simulators & reasoning systems • Status: • v1 scheduled for completion in 9/04 • Please see Matt Molineaux’s demo • 11 additional organizations about to start 1-year collaborations • Enhances probability that TIELT will achieve its goals