T.E.M.P.L.E.: Knowledge Acquisition through Extension

1. TEMPLE:TEMPLate Extension Through Knowledge Acquisition Yolanda Gil Jim Blythe Information Sciences Institute University of Southern California http://www.isi.edu/expect {gil, blythe}@isi.edu

2. Acquiring Planning Knowledge • Problem: SOF users need to add knowledge to these planning tools • ROEs, commander’s guidance • Plan evaluation/critiquing criteria • Highlight the information that is important to them • Add/extend templates • Approach: provide knowledge acquisition tools to adapt and extend pre-existing planning knowledge • Exploit ontologies and background knowledge so users don’t have to start from scratch • KA Scripts guide the user through multiple steps • Users manipulate English paraphrases of internal representations • Benefits: • Users can extend the tool’s baseline knowledge for the operation

3. Prototype for adding plan critiques: Expect’s PSM Tool User adds detailed knowledge through English paraphrases Questions formulated based on background knowledge

4. The next 100 days • Allow users to specify and customise “sentinels” that check for new information and alert planners when needed. • Our tools generate Java. • Extend ontologies and background knowledge to handle SOF domain. • Integrate with one of the jumpstart applications, probably the travel planning tool, using InterAcT.

5. Backup slides • Description of approach, tools and experiment from HPKB project.

6. Key Technologies • Guiding users through knowledge acquisition scripts [Tallis and Gil 99] that capture typical dialogues that users follow to enter new knowledge step by step • Exploiting domain-independent background knowledge about plan evaluation and critiquing [Blythe & Gil 99] that use background knowledge about plan evaluation and critiquing to guide the dialog • An English-based editor [Blythe & Ramachandran 99] that lets the user add or modify internal knowledge by manipulating its English paraphrase, without having to see or understand the internal formal representation

7. T e m p l e B a c k g r o u n d k n o w l e d g e O n t o l o g i e s ¤ c o n s t r a i n t t y p e s ¤ a c t i o n s a n d p l a n s u i s i t i o n ¤ p r o a c t i v e S c r i p t s s u g g e s t i o n s N a t u r a l E n g l i s h M e t h o d b a s e L a n g u a g e e d i t o r G e n e r a t o r S O F C o m p i l e r A c t i v e T e m p l a t e s T o o l k i t T e m p l a t e E x e c u t a b l e l i b r a r y c o n s t r a i n t s Architecture of TEMPLE D o m a i n C o n s t r a i n t ( S e r v e r ) k n o w l e d g e A c q u i s i t i o n U I ( C l i e n t ) D o m a i n C o n s t r a i n t A c q m o d e l s w i z a r d a n d t e m p l a t e s D o m a i n c o n s t r a i n t s D o m a i n m e t h o d s C o n s t r a i n t v i e w e r

8. Evaluation and Critiquing Knowledge Submethods for checking plan resources Plan ontology (PLANET) Submethods for checking plan structure Ontology of critiques Ontology of resources Reused knowledge (ontologies and methods) Domain-specific knowledge Domain-specific critiques Domain-specific plan critiquing and evaluation system Domain-specific submethods

9. isa isa An Ontology of Plan Evaluation Criteria • Captures general knowledge of how to evaluate plans with respect to standard norms of plan development Complete statement ako Statement critique Does <unit> have sufficient combat power to accomplish its mission? Correct statement ako Clear statement ako Ill-formed description Link critique isa Does the purpose of supporting effort X support the main effort? Correct link ... ako ako Structure critique Does the COA include a statement of the reserve forces? ... Complete plan

10. KA Scripts • Helps the user add new critiques by using a background theory of plan evaluation and critiquing. • KA Scripts guide the dialog with the user about the new critique (wizard-type interaction). • The tool creates some of the needed methods for the critiques, helps the user to create new ones (by suggesting initial templates), and ensures consistency with existing knowledge.

11. English-Based Editor Generates automatically English paraphrases of problem-solving fragments, and presents alternative text to replace parts of the paraphrase based on the ontologies and background knowledge NL description of method Alternatives for selected text fragment

12. First experiment: An ablation test on the PSM-Based KA Scripts • Hypothesis: the PSM-Based KA Scripts significantly reduce the expertise and the effort required to add a new critique to the knowledge base. • KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check) • Knowledge (and tool) ablation experiment: Two tasks done using PSM-Based KA Scripts, two tasks done without • Subjects: four Army officers, previously trained on EXPECT’s language for a day

13. Sample Tasks Given to Subjects • Simple critique: Add a critique to check if the COA has a security statement. • Complex critique: Add a critique to check if each task in the COA has sufficient force ratio. To compute force ratio, divide the sum of combat powers of the Blue units assigned to the task by the sum of combat powers of the Red units acted on by the task. (Two other comparable tasks were also used)

14. LEGEND: indicates total tasks Quantitative results: what users could do • Users could complete more tasks using the PSM-based KA scripts

15. Quantitative results: speed improvements • Time reduction using the PSM-based KA Scripts Time in minutes

16. Axiom acquisition rates:Experiment with PSM-Based KA Scripts with PSM-Based KA Scripts with ablated version Adding small amounts of new knowledge 1.1 ax/min 2.12 ax/min Adding larger amounts of new knowledge N/A (users were not able to do tasks) 1.26 ax/min

17. Summary • Using the PSM-Based KA Scripts significantly reduced the time taken to add a critique • Using the PSM-Based KA Scripts, all four subjects successfully added simple critiques to the EXPECT critiquer; three of them successfully added more complex critiques. • Without the PSM-Based KA Scripts, three out of four subjects successfully added simple critiques and two added more complex critiques. • Comments on the tool usability were positive in all cases.

18. Second experiment with PSM-Based KA Scripts and English-Based Editor • Hypothesis: the combination of the PSM-Based KA Scripts and English-based editor allows a user with very little training to add new critiques. • Single subject usability test: A subject was briefed in COA critiquer and the KA interface (but not about EXPECT) for 20min and asked to add two critiques using the tool • KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check), same used in the previous experiment • Subject: an Army officer with no EXPECT training

19. Results • The subject was able to add two new critiques of both low and medium complexity. • The time taken was comparable to that for the other four subjects that had previous training in Expect: Time in minutes

20. EXPECT: A User-Centered Framework for Developing KBSs EXPECT Ontologies and Method libraries Knowledge Base Method instantiator COA ontologies Domain ontologies and factual knowledge CYC/Sensus Upper Problem solving methods Evaluations and Critiques Plans (PLANET) Domain dependent KBS Resources (OZONE) Evaluation PSMs KA tools Interdependency Model (IM) KBS compiler EMeD PSMTool KA Strategies KA Scripts Knowledge-Based System Instrumentation

21. EXPECT: A User Centered Approach for Knowledge-Based Planning Tools Knowledge acquisition technology that can guide users to specify planning knowledge and develop planning tools • Expressive representations • Loom/Powerloom KR&R • EXPECT’s language to represent problem solving knowledge • Powerful reasoners • Loom/Powerloom pattern classifier & reasoners • abstract problem solving through partial evaluation • ex: how to move <a set of units> from a <location> to another <location> • Explicit models of planning knowledge and plan reasoners: • PLANET ontology of plans, OZONE resource ontology • plan evaluation and planning methods • Expectation-based knowledge acquisition tools • Derive interdependencies between individual knowledge fragments • KA Scripts to guide users in completing complex modifications