400 likes | 504 Views
Explaining Soar: Efforts to make Soar models more useful and usable. Isaac G. Councill Work conducted with Frank E. Ritter, Steven R. Haynes, and Mark Cohen This project is supported by the US Office of Navy Research, award number N00014-02-1-0021. Overview. Brief introduction to Soar
E N D
Explaining Soar:Efforts to make Soar models more useful and usable Isaac G. Councill Work conducted with Frank E. Ritter, Steven R. Haynes, and Mark Cohen This project is supported by the US Office of Navy Research, award number N00014-02-1-0021
Overview • Brief introduction to Soar • Problem statement • User study (ICCM 2003) • dTank sim (Tech. Report No. ACS 2003 - 1 ) • Herbal Viewer • Herbal IDE (23rd Soar Workshop)
Soar • Cognitive architecture / AI programming language • Production system
What is Soar used for? • Intelligent synthetic forces • Special ops training • Fixed/Rotary aircraft scenarios • Sensitivity training • Until recently, JSAF was largest training environment • Hundreds of TacAir soar agents participated with thousands of agent world wide in massive simulations
The Problem:Behavior Validation • Lives are on the line – improper training leads to mistakes in the field • Training personnel must manage 40-100 entities and ensure proper training experience • Complex agent behavior is hard to understand • Best interfaces not good enough (TacAir Soar discontinued in training exercises partly for usability reasons)
Motivation • Easier validation of agent behavior • Increased usability • Agents become easier to understand - ala Mycin • Provide better training - ala Clancey (Neomycin) • Increased potential to develop users’ trust • Enhanced understanding may enable more creative use of tools
TacAir-Soar SAP • Allows users to access and interpret internal state of agents • Supports limited view of agent operating history at run time, full off-line replay capability • Provides “what”, users must infer “why”
SAP Usability Data (Avraamides & Ritter 2002) • Expert SAP users observed while performing a set of basic SAP tasks • Participants encouraged to provide verbal explanations of agent behavior and interpret scenarios displayed by SAP • ~4 hours of video recordings of expert TacAir-Soar SAP users interacting with the SAP
Study Method • Videos transcribed and analyzed to determine where breakdowns occur in users’ attempts to understand TacAir-Soar behavior • Model/View/Controller used as framework for analysis. • Model → explanation content • View → delivery mechanisms
Results – Explanatory content • Need for explanations (14, 9%) – validates premise • Request for additional content (10, 5%) • Beyond model scope (5, 3%) • Within model scope (5, 2%)
Results – Delivery • Milestones (6, 2%) • Clarity (2, 1%), Content (4, 2%) • Goal stack (12, 8%) – disliked • Clarity (2, 2%), Content (10, 6%) • Amplified goal display (6, 3%) – new to most participants • Proposal for new delivery tool (6, 8%)
Results – Delivery (viewport) • 56 utterances, 36% of transcripts • Scale & orientation (8, 8%) • User ergonomics (6, 4%) • Scale as awareness (2, 4%) • Supplemental information (22, 11%) • Usefulness of information (16, 7%) • Requests for additional info (6, 4%) • Symbols (26, 18%) • Usefulness of symbols (12, 7%) • Requests for additional symbols (14, 11%)
Deriving Explanations A major use for SAP in applied context is to support derivation of explanations for agent behavior • Explanation derivation is currently interpretive, lacking explicit support • Viewport is most used tool for deriving explanations
Process Information • Goal stack is only conveyor of process information • Found to be useless by half of participants, of marginal utility by other half • All participants indicated incomplete understanding of goal stack • All process knowledge must be inferred by users based upon assumptions about TacAir Soar implementation
Understanding the viewport • Participants indicated confusion as to purpose of viewport – agent knowledge or real world? • Viewport displays selected declarative knowledge (but look how much) • Exploits unrealistic aspects of model • As we develop realistic models of working memory and attention, viewport-like tools may become less useful
General Conclusions • Efforts should focus on general mechanism for accessing agent knowledge and contrasting the knowledge with “real world” • Information regarding the process of agent cognition should be made more accessible
Overview • Adversarial test bed for agents and agent facilities • Provides an agent architecture-neutral interface to the game server • Humans and artificial entities built on virtually any platform can interact within the same environment over a LAN or the internet • Highly dynamic API!
Purpose • Herbal makes it easier to understand running models of Soar • Visual displays of working memory, and the PSCM task structure. • Dynamic trace views that track the history and frequency of events. • Displays that profile a running model.
Herbal is Flexible • Herbal will work with all Soar models. Nothing has to be done to the model for it to work with Herbal! • Herbal allows for the replay of previous model sessions without the need to run Soar. • Although currently designed to work with Soar, Herbal can be extended to work with many cognitive architectures, as long as the architecture is based on the PSCM.
Architecture Herbal VISTA Tcl Script Soar Monitors Soar • Herbal • Written in Java. • Uses the VISTA library to listen for event strings over a TCP/IP socket. • Soar Monitors • A collection of Soar event monitors (productions) that fire on queue and call functions located in the Tcl Script. • Tcl Script • Responds to Soar events by sending event strings to Herbal, over a socket. Soar Events
VISTA • “The Visualization Toolkit for Agents” (VISTA), developed by Soar Technology Inc. • Facilitates the creation of agent visualization applications by providing an infrastructure for communication between agents and VISTA enabled applications. • Using a communication channel, agents can convey changes to their internal state to a listening VISTA enabled application. • The VISTA toolkit also provides the ability to record and playback agent activity. • By using VISTA as the infrastructure for communication between the cognitive architecture and Herbal, a significant amount of development time was saved. *VISTA Developer’s Handbook, Soar Technology Inc., 2002.
Overview • Protégé • An objectification of Soar – the PSCM, roughly • Merging the design and programming processes • Aspect-oriented software development • Herbal IDE architecture summary
Protégé • Ontology editor – used to create object systems that describe all concepts and concept relations within a domain. • There are many plugins for protégé, including support for xml and rdf output. Protégé is a free download from http://protege.stanford.edu
An Objectification of Soar • Specified using Protégé • Constructed using PSCM as base model, with augmentations deemed appropriate • Encourages top-down design and programming
Aspect-Oriented Programming • One step beyond OOP • Programmatic behaviors often do not fit naturally discrete modules (e.g. design patterns, logging, optimization levels) • AOP allows modularization of cross-cutting behaviors in object systems
AOP Example Common solution to logging: public void doGet(JspImplicitObjects theObjects) throws ServletException { logger.entry("doGet(...)"); JspTestController controller = new JspTestController(); controller.handleRequest(theObjects); logger.exit("doGet"); }
AOP Example (cont.) Create an aspect to objectify logging and avoid repetitive coding: public aspect AutoLog { pointcut publicMethods() : execution(public * org.apache.cactus..*(..)); pointcut logObjectCalls() : execution(* Logger.*(..)); pointcut loggableCalls() : publicMethods() && ! logObjectCalls(); before() : loggableCalls() { Logger.entry(thisJoinPoint.getSignature().toString()); } after() : loggableCalls() { Logger.exit(thisJoinPoint.getSignature().toString()); } }
AOP in the Herbal IDE • Soar is difficult to program/understand/explain! • One way to slice it: The interesting parts of models aren’t just the operators they have in them, but how those operators interact with each other (aspects!) • Flexible aspect creation allows developers to specify reusable behaviors • More principled design • Better encapsulation of interesting bits • More understandable and explainable models (that’s the main point, afterall)
Resources • Protégé – http://protege.stanford.edu • RDF – http://www.dlib.org/dlib/may98/miller/05miller.html • XSLT – http://www.xml.com/pub/a/2000/08/holman/ • Aspect-oriented programming • http://www.parc.xerox.com/aop • http://aosd.net