Assisted Cognition

1. Assisted Cognition Henry Kautz Don Patterson, Nan LI Oren Etzioni, Dieter FoxUniversity of WashingtonDepartment of Computer Science & Engineering

2. Cognition in Context • Can often compensate for physical disabilities by change in environment • Wheelchairs • Redesigned appliances • Cognitive competence also depends on environment • Can you cook dinner, given a dead animal, a stone knife, and set of flints?

3. Social Context • The context for cognition involves both the physical and social environments • Stability & organization of physical environment may reduce cognitive load • Other people (e.g. a spouse) can actively assist in problem solving • How can I make coffee? • Which way is home?

4. The $80 Billion Question • Can we build computer systems that (like a caregiver) actively assist a person with Alzheimer’s perform the tasks of day-to-day living? • Enhance quality of life • Prolong aging in place • Lessen burden on other caretakers • Depression affects 20% of Alzheimer’s patients, but 50% of Alzheimer’s caregivers • Crisis in demographics – shortage of caretakers

5. The Assisted Cognition Project • University of Washington Computer Science & Engineering • UW Medical Center • Alzheimer’s Disease Research Center (ADRC) • UW Institution on Aging • Outside Collaborators: • Intel Research – Seattle and Jones Farm • OGI/OHSU • Elite Care http://assistcog.cs.washington.edu/

6. Vision Computer systems that improve the independence and safety of people suffering from cognitive limitations by… • Understanding human behavior from low-level sensory data • Using commonsense knowledge • Learning individual user models • Actively offering prompts and other forms of help as needed • Alerting human caregivers when necessary

7. Example: Activity Compass • Help user move between home and community • Walking, riding in a car, public transport • Predicts where user is going • Offers simple directions • Detects potential problems • Is user on the wrong bus? • Is user wandering?

8. Example: ADL Prompter • Joe enters bathroom at 9:00 am. • He turns on water, and picks up toothbrush. • Nothing happens for 30 seconds. AC system recognizes “tooth brushing” activity has stalled. • Prompts Joe to pick up toothpaste. Joe does so and completes task. • Joe leaves bathroom with water still running. AC system gently encourages Joe to go back and turn it off.

9. Common Architecture

11. Technical Approach • GSP equipped Palm monitors location & velocity, communicates with server • Dynamic Bayesian Net determines current mode of transportation • Learned Markov Model predicts most likely activity path – i.e., user trajectory through time and space • Each segment is a different mode of transport • History, time of day, appointment calendar, bus schedules • Guide user along activity path

12. Minimalist User Interface

13. User Feedback • User may deviate from predicted path because • System is wrong – need to update model • User is in error – confused, forgetful • System may ask for user for confirmation • “Tap if you’re okay” • Balance cost of annoying user vs. probability that user is in danger

14. Deciding When to Intervene G = prediction that help is needed (Horvitz 98)

15. Gathering Data

16. Velocity Histogram

17. Dynamic Bayesian Net BusStop B B T T+1 M M Mode T1 T1 Timing S S Speed T2 T2 Timing V V Velocity

18. Mode Prediction

20. Current Work • Measure accuracy of Markov Model for predicting activity path • Compare other approaches • Employ Relational Markov Model • Less training data • Increased power • Planning algorithms for “error correction” • E.g., once user has missed bus, find new path to achieve same goal

21. ADL Prompter • General approach: build a probabilistic model of • Common user goals • “Plans” (complex behaviors) that achieve those goals • Including failure modes • How simple behaviors are sensed • Run model “backwards” to interpret sensed data

22. Night bathroom run Get out of bed Get out of bed Walk tobathroom Walk tobathroom Walk tobedroom Get intobed Flush Location Badge Sensor Door Sensor GPS

23. Location Night snack run Get out of bed Walk tokitchen Getcrackers Walk tobedroom Get intobed Badge Sensor Door Sensor GPS

24. Night pattern Sleep Night bathroom run Night snack run Location Badge Sensor Door Sensor GPS

25. violation Night wandering Timing Constraints Night bathroom run active [9 pm – 7 am] Walk tobedroom Get intobed < 10 min

26. Summary: ADL Prompter • Commonsense knowledge base of “significant” behaviors • Hierarchically organized • Probabilistic at all levels • Several parallel ongoing activities possible • Absolute and relative timing constraints • Probabilities “tuned” by machine learning techniques for individual users • Failure modes – points of possible intervention

27. Conclusions • Growing research area combining AI, ubiquitous computing, and assistive technology • NIST, AAAI, Ubicomp Workshops • RESNA • Gerontechnology • Key idea: Patient and computer as a problem-solving team

28. End

29. Technical Foundations • Hidden Markov models • Mathematical framework for describing processes with hidden state that must be inferred from observations • Hierarchical plan networks • Represents how a task can be broken down into subtasks • Hierarchical hidden Markov models • Key to climbing food-chain!

30. Key Issue • How to go from noisy and incomplete sensor measurements to • A meaningful description of what a person is doing • “Trying to brush teeth” • “Trying to get home” • A decision by the system about whether or not to intervene … in a principled and scalable manner!

31. Interventions • Framework allows AC system to predict when a “failure” is likely • Different failures have different costs • Wandering in bedroom • Wandering outside • Forgetting to take medicine • Forgetting to flush • Must avoid:

32. Advertisement • UbiCog 2002 – Workshop on Ubiquitous Computing for Cognitive Aids • September 29, 2002 • Gothenberg, Sweden • Part of UBICOMP-2002, the major ubiquitous computing conference • Some space still available, email Henry Kautz <kautz@cs.washington.edu>

33. green – GPS readings (10 sec), yellow – location estimation (probability distribution)

35. Creating the User Model • Training Data: • 20,000 GPS readings • Predicting mode • 98% accuracy (10 FCV) • Predicting next mode transition • 97% accuracy (10 FCV)