Intelligent Online Case Based Planning Agent Model for RTS Games

1. Intelligent Online Case Based Planning Agent Model for RTS Games Ibrahim Fathy, MostafaAref, Omar Enayet, and Abdelrahman Al-Ogail Faculty of Computer and Information Sciences Ain-Shams University ; Cairo ; Egypt

2. Agenda • Introduction. • Problem definition. • Objectives. • Intelligent OLCBP Agent Model. • OLCBP\RL Hybridization . • Experiments and Results. • Conclusion & Future Work. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

3. Introduction: Online Case-Based Planning • Online Case Based Planning is an architecture based on CBR. • It addresses issues of plan acquisition, on-line plan execution, interleaved planning and execution and on-line plan adaptation. • Darmok is a previous system that applies this -without revision- to RTS Games. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

4. Introduction: SARSA(λ) Learning • A Reinforcement Learning Approach. • Combines temporal difference learning technique “One-Step SARSA” with eligibility traces to learn state-action pair values effectively. • It’s an on-policy method. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

5. Introduction: Real-Time Strategy Games • Considered a challenging domain due to: Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

6. Problem Definition • Research in learning and planning in real-time strategy (RTS) games is very interesting. • The research on online case-based planning in RTS Games does not include the capability of online learning from experience. • The knowledge certainty remains constant, which leads to inefficient decisions. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

7. Objectives • Proposing an intelligent agent model based on both online case-based planning (OLCBP) and reinforcement learning (RL) techniques. • Increasing the certainty of the case base by learning from experience. • Increasing both efficiency and effectiveness of the plan decision making process. • Evaluating the model using empirical simulation on Wargus. ( A clone of the well-known strategy game Warcraft 2) Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

8. Agent Architecture – Abstract View Environment ( RTS Game “Wargus”) Actions Traces Offline Phase (Case Acquisition) Plan Expansion/Execution Module Online Case-Based Learner Behaviors Retrieved Case Case Base Cases Online Phase Goals Evaluated Case Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

9. Agent Architecture – Detailed View Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

10. Case Representation Case Goal Strategy Situation State Shallow Features Deep Features Behavior Pre-Conditions Alive-Conditions Success-Conditions Snippet Learning Parameters Certainty Factor Eligibility Prior Confidence Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

11. Case Representation - Example Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

12. Case Retrieval • Since the evaluation of the case base is changed, the case retrieval algorithm must change also. • The case with the best predicted performance will be retrieved to be executed. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

13. OLCBP/RL Hybridization Algorithm RL Algorithm used: SARSA(λ) Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

14. OLCBP/RL Hybridization: The Mapping.

15. Experiment: Case Study Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

16. Results of Attack1 & Attack2 Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

17. Results of BuildArmy1 & BuildArmy2 Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

18. Results (Cont’d) • Agent has learnt that building a smaller heavy army in that specific situation (the existence of a towers defense) is more preferable than building a larger light army. Similarly, the agent can evaluate the entire case base and learn the right choices. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

19. The Paper –Conclusion • Online case-based planning was hybridized with reinforcement learning in order to introduce an intelligent agent capable of planning and learning online using temporal difference with eligibility traces: Sarsa (λ) algorithm. • The empirical evaluation has shown that the proposed model –unlike Darmok System - increases the certainty of the case base by learning from experience, and hence the process of decision making for selecting more efficient, effective and successful plans. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

20. The Paper –Future Work • Implementing a prototype based on the proposed model. • Developing a strategy/case base visualization tool capable of visualizing agent’s preferred playing strategy according to its learning history. This will help in tracking the learning curve of the agent. • Finally, designing and developing a multi-agent system where agents are able to share their experiences together. Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10

21. Thank You ! Questions ? Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10