Enhancing Interactions with To-Do Lists Using Artificial Assistants

1. Enhancing Interactions with To-Do Lists Using Artificial Assistants Yolanda Gil, Timothy Chklovski USC/Information Sciences Institute {gil, timc}@isi.edu March 26, 2007

2. Learning Common Knowledge from Volunteers to Support Assistance Learner Learning about objects Learning about tasks Validates through other users Guides knowledge entry Formulates relevant followup questions in real time Learning paraphrases Proactively broadens coverage Learning to anticipate and repair Targets collection by topic and knowledge type Structured statements about tasks and objects: - setting up a videoconference • camera:switch on • computer:turn on, start up, link up • microphone:turn on, test, adjust Problems & remedies: - during a meeting • projectornot available • locate a portable projector 600,000+ statements collected over 12 months

3. To Do Lists • To Do lists are pervasive, and present large opportunity for assistance and learning • We’ve been working with TOWEL, the To Do list manager in CALO • Glimpses / hints of users’ goals • To Do lists • Have some regularity & structure • Contents and surface form may vary widely • Similar to the collected statements

4. To Do Lists: Related Work • Ethnographic studies look at usage of To Do lists • Eg: V Bellotti, B Dalal, N Good, P Flynn, D Bobrow, N Ducheneaut.What a To-Do: Studies of Task Management Towards the Design of a Personal Task List Manager. CHI 2004 • Analysis of work activities and how tools may support it, cognitive aids • B Harrison. An Activity-Centric Approach To Context-Sensitive Time Management. In CHI 2004: Workshop on the Temporal Aspects of Work. • A Dey, G Abowd. CybreMinder: A Context-Aware System for Supporting Reminders. HUC, 2000 • Some commercial tools support prioritization • Based on activity type, urgency, time available • Eg, Life Balance, http://www.llamagraphics.com/, Voo2Do, “Remember the milk”, Tada.com (provides public to do lists) • Not focused on NL interpretation or on automation, mostly on human factors and usability

5. Assisting with To Do Lists: the Idea • Key idea: Develop interpretation & mapping of To Do entries to assistant-supported tasks • Exploit large knowledge repositories and preprocessed texts • Paraphrases to help interpret text • Use knowledge repositories to interpret and connect user knowledge to operationalized tasks in CALO • Build on prior work on volunteer collection of paraphrases to assist speech recognition / utterance identification [Chklovski, KCAP ’04 & KCVC ’04] • Build on prior work on extraction from large corpora [Chklovski & Pantel, EMNLP ’05], and volunteer validation

6. BEAM: “Broad-coverage Entity Analysis and Mapping” • To Do interface integrating BEAM and providing task interpretation and monitoring via CALO’s TOWEL • List is automatically updated when the “Plan conference travel” action is completed To Do entry made by user BEAM mapping to TOWEL task

7. Opportunities for Interpretation-based Assistance User To Do Entries CALO Task Ontology: Catalog of Automated Procedures BEAM Find hotel w/ pool for Joe Map entries to task procedures Reserve conf room for talk Reserve accommodations Buy bread on the way home Anticipate & suggest missing Entries, sub-tasks Host a visitor Get talk abstract from Joe Group and organize entries Arrange meeting Announce room for talk Initiate and report execution Execution and Monitoring Task Learning Set airport pickup for Joe Assist with how activities are done in the organization Instrumented Desktop

8. BEAM Knowledge Sources for NL Interpretation and Assistance From Web Volunteers From Text Extraction From Knowledge Engineers From Volunteers in the Organization Verb Relations Repository: “schedule happens-beforereschedule” Repairs Repository: “If projector not working, try a new bulb” NGramsRepository “to schedule a meeting” Action Paraphrases Repository: “plan X  schedule X” “lease car  rent car” Subtasks Repository: “Reserve X has-subtask find X” Ontologies Organization-specific Task knowledge “Airport pickup of visitors is common, but not here” User To Do Entries CALO Task Ontology: Catalog of Automated Procedures BEAM Find hotel w/ pool for Joe Map entries to task procedures Reserve conf room for talk Reserve accommodations Buy bread on the way home Anticipate & suggest missing Entries, sub-tasks Host a visitor Get talk abstract from Joe Group and organize entries Arrange meeting Announce room for talk Initiate and report execution Execution and Monitoring Task Learning Set airport pickup for Joe Assist with how activities are done in the organization Instrumented Desktop

9. BEAM Knowledge Sources for NL Interpretation and Assistance From Web Volunteers From Text Extraction From Knowledge Engineers From Volunteers in the Organization Verb Relations Repository: “schedule happens-beforereschedule” Repairs Repository: “If projector not working, try a new bulb” NGramsRepository “to schedule a meeting” Action Paraphrases Repository: “plan X  schedule X” “lease car  rent car” Subtasks Repository: “Reserve X has-subtask find X” Ontologies Organization-specific Task knowledge “Airport pickup of visitors is common, but not here” User To Do Entries CALO Task Ontology: Catalog of Automated Procedures BEAM Added by user Find hotel w/ pool for Joe Map entries to task procedures Reserve conf room for talk Reserve accommodations Marked by BEAM as user-only Buy bread on the way home Anticipate & suggest missing Entries, sub-tasks Host a visitor Executed & monitored Get talk abstract from Joe Interpret entries Group and organize entries Arrange meeting Added by BEAM Announce room for talk Tolerate syntactic variety Initiate and report execution Execution and Monitoring Task Learning Deleted by BEAM because unnecessary Identify Omitted Actions Set airport pickup for Joe Assist with how activities are done in the organization Instrumented Desktop Identify Related Sub-tasks

10. Our Approach • Extended existing semantic parsing techniques to take advantage of broad-coverage knowledge repositories • Based on the standard Semantic Parsing approach [Acero; Zue; Allen et al] • Syntactic chunking + identification of categories of entities + semantic parsing of annotated result • Our problem is simpler in some ways • Allows simplifying assumptions about structure of entries, speech acts present • Our problem is harder in other ways: actions may not be fully specified, new actions may be automated (learned) • To assist with user requests, need to identify implied actions & sub-actions • Leverage large knowledge repositories • Leverages paraphrase collection for speech system [Chklovski, K-CAP05, KCVC-05]

11. BEAM in Year 3: What Can and Cannot Be Interpreted • Statements are processed into their semantic components • Some entries cannot be interpreted because content is not recognized • Paraphrasing knowledge allows rewriting of entries so they can be interpreted

12. BEAM’s Stages of Mapping a User’s To Do Entry Teraword textual frequency repository 1. Syntactic parsing 2. Identification of semanticcomponents, present & implied User’s To Do entry knowledge from volunteers and text extraction 3. Ontological mapping of semantic components 4. Automatic entry rewriting ontology Volunteer contributed paraphrases (also task-subtask pairs) yes Mapped? Rewrites available? no no yes Mapped entry Mapping failed

13. BEAM’s Stages of Mapping a User’s To Do Entry Teraword textual frequency repository 5. Identification of sub-tasks 1. Syntactic parsing 2. Identification of semanticcomponents, present & implied User’s To Do entry knowledge from volunteers and text extraction 3. Ontological mapping of semantic components 4. Automatic entry rewriting ontology Volunteer contributed paraphrases (also task-subtask pairs) yes Mapped? Rewrites available? no no yes Mapped entry Mapping failed

14. Using BEAM with a To Do Manager: BEAM API

15. Recent Developments: Smarter BEAM Identifies Likely Implied Actions To Do entry does not specify an action BEAM looks in 1012 word corpus for mentions of “to * a meeting”, etc, identifying actions These actions are then mapped to the specific target ontology http://seagull.isi.edu/cgi-bin/todo-mgmt/api2?q=a%20meeting%20with%20Yolanda;gv=1

16. Leveraging the TerawordNgrams Source • “quarterly meeting on Monday”  “to * a meeting” • “presents for John”  “to * a present” • 4876 buy 642 send 3482 get 619 bring … 593 open 1751 give 499 wrap 1352 find http://seagull.isi.edu/cgi-bin/ngram-extract/query-ngrams.pl • But what if there is little or no data? (Eg, “TGW meeting”) • We’re exploring backoff strategies • But what if there is noise in the data? • Stoplists (eg, “be”, “have”) can provide some relief • Validation & feedback could also help

17. Example of Volunteer-based Validation Completed in 2006 Snapshot of validation: • 1,113 harvested statements were put through context-directed validation for “likely to be purchased in an office categories of times” • Most were filtered out; 107 (9.6%) passed Determined to belikely to be purchased in an office:0.977        'office supplies' > planners 0.922        'office supplies' > equipment0.674        Business & Industrial > Food Service & Retail > Bar & Beverage Equipment > Coffee Categories determined to be unlikely to be purchased in an office: 0.017       Home & Garden > Pet Supplies > Cats > Cat Toys0.000       'building supplies' > 'concrete finishing'0.000       cards > 'racing-nascar'

18. Towards Identifying Subtasks:Start with Large Corpus • “a meeting with Peter” • QUERY1: “the [Y] for the [X]” • QUERY2: (filter) “need the [Y]” • QUERY3: “to [Z] the [X] [Y]” • “approve meeting agenda”, • “set/change/confirm meeting time” • Similarly: for “flight to SFO” • Top suggestions include: • “buy/purchase flight ticket”, • “make flight arrangements”

19. Towards Identifying Subtasks:Start with Large Corpus • “a meeting with Peter” • QUERY1: “the [Y] for the [X]” • QUERY2: (filter) “need the [Y]” • QUERY3: “to [Z] the [X] [Y]” • “approve meeting agenda”, • “set/change/confirm meeting time” • Similarly: for “flight to SFO” • Top suggestions include: • “buy/purchase flight ticket”, • “make flight arrangements” Under Development – Stay Tuned

20. 2006 Test System: Mappings to Task Ontology Categories • Early integration done to support a test question not otherwise addressed by CALO: • Any small contribution to test results considered a success • No lead time for targeted collection for relevant task entries PQ0166: What instances of task type (choose one) {|sc:%Communicate|, |sc:%Decide|, |sc:%Obtain|, |sc:%PlanAndSchedule|} are on user’s to-do list? • Examples handled by BEAM: • arrange travel #PlanAndSchedule::Plan • office supplies #Obtain::Buy • hire a car  #Obtain::Rent • respond to Mary's request #Communicate::Answer • This is more powerful than straightforward application of, eg, WordNet synonyms • BEAM handles some situations where there is no synonymy or is-a relation between terms, eg. (arrange travel #PlanAndSchedule::Plan) • BEAM handles some situations where action is not even present (office supplies #Obtain::Buy)

21. Evaluation of Paraphrase Component • Despite early integration, BEAM contributed to evaluation • 582 To Do statements collected for CALO Y3 evaluation • 31.1% were mappable using the paraphrase repository • 24.6% without paraphrase repository • Paraphrase repository was collected without focus on these items specifically • Contained 3,114 items, but (we estimate) only ~100 related to the domain covered by the test question • Additional knowledge sources and larger repositories will support further improvement of performance • Now also have data from public online To Dos, “tada list”

22. Conclusions • Developed BEAM, first system to demonstrate To Do list interpretation to enable automation • Integrated with working ToDo list manager, CALO’s Towel • Extended existing semantic parsing techniques to take advantage of preexisting large knowledge sources • Leveraged volunteers-created paraphrase corpus to improve ToDo entry interpretation • Identify likely implied actions • Subtask suggestion in the works

23. Ongoing & Planned Work: New Capabilities • Improve interpretation and mapping capabilities: • Evaluate support of To Do entries which have no verb – ability to identify the implied actions • Proactively identify automatable subtasks for To Do lists • Acquire knowledge about relevant subtasks • Use BEAM’s semantic frames to provide information for task arguments, Towel forms (eg, “travel to Boston”) • Validate with/acquire from volunteers knowledge about sub-tasks, mappings To Do entry made by user BEAM mapping to TOWEL task