Interactive Knowledge Capture for Problem-Solving Systems

1. Interactive Knowledge Capturefor Problem-Solving Systems Jim Blythe Yolanda Gil Jihie Kim www.isi.edu/ikcap

2. Outline • Main elements of our approach • Experiences and lessons from earlier work: monolithic systems • New work on the Calo project: open systems

3. Acquiring procedure knowledge from users • Each user can have unique requirements of a system • Their requirements will change over time, perhaps frequently • In travel planning, may be different for each trip • Users need to be able to modify procedure representations in intelligent systems to address their needs • Complements example-based learning

4. Example: adding procedure knowledge to a travel planning tool • An assessment tool makes judgments about travel itineraries: • e.g.,the airline should be United or American • e.g.,the hotel should be within walking distance, unless I am renting a car • Need to add procedure knowledge to tell the system to make a new kind of judgment: “the hotel can cost up to 20% more than the government per diem rate for the city.” or supporting procedures: “to estimate driving time, divide the distance by 55”

5. Expect’s Support for procedure KA:Key Technologies • Where does the user start? • An acquisition wizard guides the user to start the KA process through a dialog, based on problem-solving methods. • KA takes many steps; users will be lost… • The acquisition wizard manages the process from end to end. • Users don’t know the computer language. • An English-based procedure editor • Users modify the English paraphrase of the formal representation. • A search-based expression composer • Suggests valid reformulations of user sentences. • How do users ensure all the needed information is added? • An interdependency analyzer understands which pieces of knowledge are used to solve a problem.

6. Main elements of our approach • Meta-reasoning: • About how knowledge fragments are combined to solve a goal • About missing information, effects of changes to KB • About the dialog and context • Putting knowledge in the user’s terms: • Generate text descriptions to hide the syntax • Browse-and-replace interface for procedures • Search-based reformulations of free text • Test knowledge on real examples

7. Outline • Main elements of our approach • Experiences and lessons from earlier work: monolithic systems using Expect • New work on the Calo project: open systems

8. Acquisition wizard • Dialog with user to start the process. • Some questions use menus or text input. • Others use the English editor to refine procedural knowledge

9. English-based Procedure editor (multiply (obj (look-up (obj fsa-per-diem-hotel-rate) (for (r-city ?hotel)))) (by 1.2)) NL description of method Alternatives for selected text fragment

10. Test new knowledge immediately Softools, developed under the DARPA AcT program Each element is defined and checked using constraints

11. Global Procedure Analysis: Test interdependencies, both rule-rule and rule-data Problem-Solving Methods Domain Ontology ... • (evaluate (obj coa) • (wrt logistics)) r-location port INTERDEPENDENCIES seaport r-berths airport r-pols ... ... (r-location port) r-piers inland waterway seaport maritime seaport r-storage-area ... (r-berths seaport) U: new port: Havana S: I need to know if it is an airport or a seaport U: seaport S: I need to know the location and the berths Interdependencies guide Knowledge Acquisition

12. When combined, the tools use information from each other Application Acquisition wizard Interdependency analyzer Acquisition analyzer Procedure editor Expression composer Instance editor Relation/concept editor

13. The acquisition wizard • Guides the user through the initial steps of adding new knowledge. • Structures the knowledge to be added using default procedural knowledge. • Questions are generated from a problem-solving theory.

14. Problem-solving theory for plan evaluation • A hierarchy of generic types of plan judgments with default procedural knowledge. judgment global judgment local judgment bounds check extensional check upper bound lower bound positive negative completeness judgment hotel cost judgment “Warn if the value is too large?” DEFINED: check that the value is less than the maximum value ASK USER: compute a maximum value for each object

15. Benefits of integration: The acquisition wizard and the method editor • Each component receives information from the other that helps the user: • The wizard provides to the editor: • An initial version of the method, with the correct capability • An expectation of the result type of the method • The editor provides to the wizard: • A more detailed method result type • Used to help classify the new task in the ontology

16. bounds check upper bound lower bound “Warn if the value is too large?”

17. The expression composer • Assists users in formalizing informal statements by suggesting composite expressions using terms the system understands. • Uses an ontology of terms and relations, and synonyms derived from WordNet with extensions. • Makes breadth-first forward search, matching keywords

18. Expression composer example User types: “max staging post landing” Tool suggests: “find the maximum of the landing distance available of the runways of the forward staging post”. Function call find object of reformulation landing-distance-available maximum Typed variable runways combining queries ?forward-staging-post Information element

19. Integrated with the method editor • Constructs compound terms in KB that include user terms and have desired type • Anytime breadth-first search through space of terms

20. Outline • Main elements of our approach • Experiences and lessons from earlier work: monolithic systems • New work on the Calo project: open systems

21. ACT2 Cue: (TEST (source vision new)) ACT1 Cue: (ACHIEVE (satisfy source)) (ACHIEVE (derive IN-list source)) ACT1 current Tailor: interactive task acquisition forthe Calo office assistant • A personal assistant that manages everyday tasks. • Learns from experience, interacts with the user naturally through several channels. • Large DARPA-funded project run by SRI. New Facts & Goals 2 (source commanders-vision new) 7 (ACHIEVE (satisfy Incoming-INs)) Process Library • Spark is Calo’s task manager. (A PRS-based system developed by Myers et al.) 1 Process Execution Facts & Goals 6 External World 8 5 Goal2 Goal3 ACT8 ACT3 3 sleeping sleeping Intention Graph Fact1 ACT2 4 normal

22. Approach and challenges in acquiring knowledge within CALO • Carry key elements forward: • Explicit reasoning about knowledge dependencies and acquisition context • Expressing the procedures in the user’s terms • Earlier work used Expect, a monolithic system using custom problem-solving system, language, methods, ontology,.. • Now use Spark’s performance element and procedures, ontologies provided by several groups.

23. Example: purchasing a laptop • Spark manages the workflow as the user purchases a laptop. • Choose a model, find bids, get authorization, track purchase… • During the process, Spark finds that the order cannot be completed, because a manager who must authorize the purchase is not available. • User should be able to tell the system “You don’t need authorization when the cost is less than $2000”

24. Initial solution • Make direct analysis of the Spark procedures • Encapsulates a model of Spark’s behaviour • Requires additions to language, e.g. types, purpose of subtasks • Combine the procedure analysis with the expression composer to help interpret user sentences. • Dialog is currently implicit • Currently provide follow-up questions based on analysis.

25. Response to“You don’t need authorization when the cost is less than $2000” Automatic generation of text from Spark procedure definitions Expression composer suggests valid condition User can explore modifications based on different assumptions

26. Tailor: summary and future work • Currently allow users to modify existing procedures, using global analysis and expression composer. Next: • Support defining new tasks, integrate with advice for Spark. • Model dialog: build templates for supporting theory, working with Allen/Ferguson on dialog model.

27. Summary • Our work allows users to add procedure knowledge to both custom and pre-existing intelligent systems. • Combination of global analysis and term reformulation can allow users and the system to reach shared understanding. • Interactive knowledge capture complements example-based learning to allow intelligent systems to adapt. • Evaluation strategies: • Ablation user studies with fixed and free tasks. • Good results with Expect & Constable, still to do with Tailor.