1 / 38

Adaptive Intelligent agent in real-time strategy games

Adaptive Intelligent agent in real-time strategy games. A Hybrid Online Case-Based Planning &Reinforcement Learning Approach. Project Members. Omar Enayet. Abdelrahman Al- Ogail. Ahmed Atta. Amr Saqr. Dr. Mostafa Aref. Dr. Ibrahim Fathy. Agenda. Project Research Area & Domain

lupita
Download Presentation

Adaptive Intelligent agent in real-time strategy games

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Adaptive Intelligent agent in real-time strategy games A Hybrid Online Case-Based Planning &Reinforcement Learning Approach

  2. Project Members Omar Enayet Abdelrahman Al-Ogail Ahmed Atta AmrSaqr Dr. MostafaAref Dr. Ibrahim Fathy

  3. Agenda • Project Research Area & Domain • Motivations. • Problem Definition • Objectives • Related Work • Our Methodology. • Offline Stage. • Online Stage. • Testing and Results. • Conclusion and Future Work. • Demo. • References.

  4. Project Research Area AI Planning Knowledge Sharing AI Learning Plan then re-plan according to new givens. Let everyone know instantly what you knew through experience. Make the machine learn.

  5. Project Domain RTS Games Real-Time Strategy Games. Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions

  6. Motivations War Simulation Experimental Relevance Robotics For the corporation of robots. They constitute well-defined environments to conduct experiments. For interest for military which uses battle simulations in training programs.

  7. Problem Definition Experience Loss Static Scripts Computer AI relies on static scripting techniques. The Absence of sharing experience costs a lot

  8. Problem Challenges Predictability Non-Adaptability Computer Opponent actions easily predicted. Computer Opponent doesn’t adapt to changes in human actions.

  9. Objectives Mobile Experience Adaptive A.I. Making the Computer Opponent adapt to changes like human do. Import/Export your experience !

  10. Related Work • SantiOntanon introduced Darmok 2 which is an Online Case-Based Planningsystem designed to play Wargus in 2010. • Eric Kok introduced : Adaptive Reinforcement Learning Agents in RTS Games, which merged BDI Agents technology with Reinforcement Learning, 2009 • M.Johansen devised a CBR/RL system for learning micromanagementin real-time strategy games, 2009

  11. Architecture Overview

  12. Case Behavior Goal WinWargus Preconditions UnitExist(Peon) Strategy Rush-Attack Alive conditions Situation PlayerBuildings > 0 Beginning Snippet Shallow Features BuildBase1 TrainForce(TinyLandForce) TrainForce(TinyAirForce) Attack(NearWoodPeon) NumberOfPlayerPeons = 10 NumberOfPlayerCanonTower = 3 PlayerHasFortress = 1 . . Performance Deep Features 0.6 PathExists = 1 DistanceToEmeny = 40 . . Eligibility Prior Confidence 3 0.8 Case Representation

  13. Perception

  14. Game State Analyzer

  15. Offline Stage: Learning from human demonstration before shipping game

  16. Case Abstraction

  17. Case Abstraction (Cont’d) • Simplify case complexity. • Increase the flexibility. Abstractor Point(10, 137) Unguarded Region

  18. Case Acquisition

  19. Case Acquisition (Cont’d) • Generates Cases from Human’s game play. Acquisition Abstract Trace Casebase

  20. Online Stage: Learning what’s the best to do while playing

  21. Case Retrieval

  22. Case Retrieval – Cont’d Case Case Base Retriever Goal

  23. Case Adaptation

  24. Case Adaptation(Cont’d) • Adapts Behaviors according to current game state. • Removal of unnecessary actions. • Adaptation for unsatisfied preconditions. Adaptation Adapted Behavior Behavior

  25. Online Plan Expansion & Execution

  26. Online Plan Expansion & Execution (Cont’d)

  27. Action Controller

  28. Case Concretization

  29. Case Concretization (Cont’d) • Adapt the abstract actions to suit current situation. Concreter Unguarded Region Point(10, 137)

  30. Case Revision

  31. Case Revision (Cont’d) Uses reinforcement learning, TD-learning SARSA(λ) Case Reviser Evaluation Used Case

  32. Testing and Results

  33. Conclusion • A Hybrid Architecture of case based reasoning and reinforcement learning was introduced to play strategy games. • The architecture merged online case based planning with Sarsa(λ) with eligibility traces. The system showed promising simulation of human behavior; however it still needs a lot extra effort and testing to become industrially capable. • Also, the concept of an abstract case base was introduced which opens the door for generic AI engines for games which is never implemented till the date of writing of this document.

  34. Demo!

  35. Future Work 1) Cooperative AI Agents. 2) Opponent Modelling. 3) Strategy visualization tool. 4) Generic situation assessment. 5) Learn weights of Game State through neural network. 6) Online I-Strategizers. 7) Generic Abstraction/ Concretization.

  36. Web Resources • To get introduced for the whole project journey, evolution, technical summaries, presentations, discussions, meeting minutes and others visit project blog: http://rtsairesearch.wordpress.com/ • For full materials of papers, technical summaries, documentations, articles, external links and running version of WARGUS (our test best) use the repository link: svn checkout http://rtsairesearch.googlecode.com/svn/trunk/ rtsairesearch-read-only • For downloading the latest source code for the I-Strategizer Project, please use the following: svn checkout http://istrategizer.googlecode.com/svn/trunk/ istrategizer-read-only

  37. References • [1] Martin Johansen Gunnerud. A CBR/RL system for learning micromanagement in real-time strategy games. In Norwegian University of Science and Technology, 2009 • [2] SantiOntañón, NehaSugandh, KinshukMishra, Ashwin Ram. On-Line Case-Based Planning. In Computational Intelligence, 26(1):84-119, 2010. • [3] Brain Schwab. AI Game Engine Programming. Charles River Media, 2009. • [4] SantiOntañón and Kane Bonnette and PrafullaMahindrakar and Marco A. G´omez-Mart´ın and Katie Long and JainarayanRadhakrishnan and Rushabh Shah and Ashwin Ram. Learning from Human Demonstrations for Real-Time Case-Based Planning. In AAAI 2008 • [5] Ralph Bergmann and Wolfgang Wilke,On the role of abstraction in case-based reasoning

  38. References – Cont. • [6] KinshukMishra, Santiago SantiOntañón, and Ashwin Ram. Situation Assessment for Plan Retrieval in Real-Time Strategy Games. In 9th European Conference on Case-Based Reasoning (ECCBR 2009), Trier, Germany. • [7] NehaSugandh and Santiago SantiOntañón and Ashwin Ram . On-Line Case-Based Plan Adaptation for Real-Time Strategy Games. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008) • [8] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, An Introduction. MIT press, 2005. • [9] Wikipedia, the free encyclopedia. http://www.wikipedia.com • [10] Michael Buro, Call for Research in RTS AI, AAAI 2004

More Related