Task Learning by Instruction: Benefits and Challenges for Intelligent Interactive Systems

1. Jim Blythe, Prateek Tandon and Mandar Tillu USC Information Sciences Institute Task Learning by Instruction:Benefits and Challenges for Intelligent Interactive Systems

2. Overview • Complex tasks in intelligent assistants • Overview of task learning by instruction in Tailor • Integration with other task-related capabilities • Challenges and new approaches in specifying tasks

3. Handling complex tasks in an intelligent assistant Medication Regime Travel Assistant UI Task models Domain models Contracts Vacation Activities

4. Specialized capabilities for complex tasks Application User invocation Advice/ Preferences Explanation Monitoring Trouble-shooting Autonomy Opportunism/ Synergy …. Assistant UI Plan creation Learning/ Instructability Sharing tasks and goals Task models Domain models

5. Specialized capabilities for complex tasks Application User invocation Advice/ Preferences Explanation Monitoring Trouble-shooting Autonomy Opportunism/ Synergy …. Assistant UI Plan creation Learning/ Instructability Sharing tasks and goals Task models Domain models

6. To purchase equipment: 1. find two one bids 2. fill out form 10C 3. if it costs > $500 request authorization from two managers 3½. record purchase on web site 4. if authorization granted place order with purchasing Modifying a procedure with Tailor:“Task Learning by Instruction” User enters text instruction, e.g. “You don’t need authorization when the cost is below $2000.” Tailor interprets by (1) recognizing type of modification, (2) mapping terms to its task and world ontology, (3) analyzing the proposed modification. {defProcedure PurchaseEquip cue: [do: (purchase $obj_id)] precond: (and (user $user) (pSelId $obj $obj_id)) body: [seq: [do: (find_bids $obj $bid)] [context: (use_form $form $obj) do: (complete_form $form $obj)] [select: (and (total_price $bid $price) (< $price 2000)) [do: (get_authorization $obj $user)]]]} To purchase equipment: 1. find two bids 2. fill out form 10C 3. request authorization from two managers 4. if authorization granted place order with purchasing

7. Modifying a procedure with Tailor compared with learning from observation/demonstration Benefits: Challenges: • Users directly describe the desired change • Users describe steps without taking them • Users can ignore implementation details • Map user instruction to procedure knowledge using context (e.g. other intentions) • Navigating a set of complex alternatives • Follow-up questions to resolve ambiguity and help fix introduced problems [Blythe, IUI 05; AAAI 05]

8. User describes a new taske.g. “list hotels within +miles of +meeting”

9. The first step “List each hotel name”

10. Aggregating DP approach to find compositions of tasks User instruction: “you don’t need form B when the seller is in the US” Tailor finds matching expression: (and (vendor $selection $v) (country $v US)) Shown as: “when the country of the vendor of the selected laptop is the US” Synonym from WordNet: seller Variable and description from GPA laptop Vendor 0:$selection (GPA) “the selected laptop” company 0: 1:(Vendor $sel’n $v+) Object from user instruction HasDisplay ComputerTotalPrice, ShippingCost, RAM, … Country display country DialCode 0: 1:(HasDisplay $sel’n $d+) Size 0:US (constant) 1: number match 0: 1:(Price $sel’n $p+), .. 2: (and (HasDisplay $sel’n $d) (Size $d $s)), .. 2:(and (Vendor $sel’n $v) (Country $v US)) <, >, …

11. Recent improvements: extend DP with dynamic grouping of partial matches • Group partial expressions by datatype and set of instruction words matched • invariant wrt. propagation and matching • Propagate groups, not individual expressions • Exponential reduction in propagations, and in iterations required to find a match Vendor laptop company Country 0:$selection (GPA) “the selected laptop” 0: 1:(Vendor {lapt: Ø}) country $selection ε {laptop: Ø} 0:US (constant) 1: (Vendor {laptop: Ø})ε{company: seller} match 2:(and {cmpny: seller} (Country ? US)) …ε{country: seller, US}

12. The last step and resulting procedure

13. Overview • Task learning in desktop systems • Overview of task learning by instruction • Integration with other task-related capabilities • Challenges and new approaches in specifying tasks

14. Challenges highlighted by Tailor prototype • Users unsure of the “next step” • More context-dependent menus and wizards • Fewer options, although there are several valid alternatives • Many alternative input sets for primitive procedures • Full list is overwhelming: be selective and summarize • Constraints on inputs of primitive procedures • Procedure hard to understand • Combining tasks built by users – inappropriate information • Hide substeps • Hide parameters • Allow drill-down

15. Task-related capabilities: communication and reasoning Communicating with user Reasoning Plan creation Monitoring Learning by observation Opportunism Collaboration Invocation Explanation Advice/ preferences Learning by instruction/ demo Autonomy

16. Context-related guidance Select modification: Add step Add input Done In order to find available hotels near the meeting with inputa meeting: No steps currently defined Add a step

17. Hiding steps and parameters Describe step to add: Hide parameters: focus on distinct actions get hotels OK In order to find available hotels near the meeting with input a meeting: get hotel chains with frequent flyer miles for American Options get hotel chains with frequent flyer miles for American get hotel chains with frequent flyer miles for United get hotel chains with frequent flyer miles for Delta get hotels from onlineReservationz

18. Heuristic: show external actions (e.g. info access), hide internal book-keeping Describe step to add: list hotel if distance < 2 OK Highlight new inputs that are automatically suggested In order to find available hotels near the meeting with input a meeting, a street and a zip: get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Options Options Show steps that determine information access

19. Show more detail about steps when the user explores the alternatives Describe step to add: list hotel if distance < 2 OK In order to find available hotels near the meeting with input a meeting, a street and a zip: Show parameters when the user asks to see alternatives get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Options Options distance between the street and zip, and the hotel street and zip options that don’t use the hotel street and zip options that don’t use new input variables Explore only through the inputs, not how they are used

20. Reduce search space using constraints on inputs Given 3 locations: YahooDistance(street1, zip1, street2, zip2):81 alternatives Given: ‘location’ = tuple(street, zip) YahooDistance(loc1, loc2):9 alternatives Given: output is a distance metric: symmetric, d(X,X) = 0 YahooDistance(loc1, loc2):3 alternatives

21. Constraints may be reusedfrom task ontology Action Information-generating action Symmetric: only consider one ordering constraints Information-generating action: output includes a distance between two inputs A and B Identity: d(A,A) = 0, so ignore B = A. YahooDistance(A:loc, B:loc) -> distance_miles loc = compound(street, zipcode)

22. High-level organization of outputs Select modification: Done Output is created in list form because the constructor is embedded in an iteration In order to find available hotels near the meeting given a meeting, a street and zip: get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Output is the list of hotels Add step before this Add test Remove this step Change step inputs Mark step as output Show details

23. Result: simpler description and control Select modification: Done In order to find available hotels near the meeting with input a meeting, a street and zip: get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list Output is the list of hotels Instead of

24. Current work • Creating primitive procedures while creating top-level procedure, not before. • Using examples from previously-build procedures to select steps, bindings and follow-ups • Constraints: representation, input and use

25. Summary • Learning complex activities from instruction. Similar issues in modules that communicate about tasks • Many potential matches for an instruction • Describing learned tasks: find the ‘right’ level of initial detail and allow drill-down • Utilize constraints on possible procedure inputs