Technologies to Improve Quality of Life for People with Cognitive Disabilities

1. Technologies to Improve Quality of Life for People with Cognitive Disabilities Henry Kautz Department of Computer Science University of Rochester Rochester, NY

2. Assisted Cognition Systems Henry Kautz Department of Computer Science University of Rochester Rochester, NY

3. Vision • Understanding human behavior from sensor data • Actively prompting, warning,and advising • Alerting caregivers as necessary Computer systems that improve the independence and well-being of people suffering from cognitive disabilities by…

4. Foundations • Pervasive sensing • Movement, location, object manipulation, physiology • On user or embedded in the environment • AI / machine learning • Model everyday life • Adapt to individual's behavior and goals • Human computer interaction • Keep focus on user • Ease of use

5. Outline • Designing an adaptive wayfinding system for cognitively impaired users • Creating a context-aware task prompting system • Challenges in research on assisted cognition systems

6. Outline • Designing an adaptive wayfinding system for cognitively impaired users • U Washington Computer Science • Alan Liu, GaetanoBorriello • U Rochester Computer Science • Henry Kautz • U Washington Rehabilitation Medicine • Kurt Johnson, Mark Harniss, Pat Brown • Creating a context-aware task prompting system • Challenges in research on assisted cognition systems

7. Wayfinding Difficulties • Alice doesn’t go to new places by herself because even with practice, she still has trouble remembering directions. • Bob gets lost easily, and cannot use a map to orient himself and figure out where he is or where to go. • Carol takes a regular route between home and work, but one day a road along the way is closed and she doesn’t know how to adapt.

8. GPS Systems • Why aren't regular GPS systems adequate? • "One size fits all" route planning • Minimize distance or time when driving • "One size fits all" presentation of directions • Map and turn-by-turn directions • What makes a route safe and easy to follow by a cognitively impaired user?

9. Research Challenge • Individuals with cognitive impairment can have vastly different abilities and disabilities • Designing for a "universe of one" (Kintsch & DePaula) • However: • Can we uncover common features that lie behind successful solutions? • Can we incorporate these features into a general system that adapts to the individual?

10. Research Path • Formative usability studies • Studied direction-following capabilities and priorities • Potential users interacting with prototypes in situ • Landmark-based directions • Incorporated landmarks into wayfinding • Balancing cognitive effort and attention to environment • Modeling individual variation • Implemented framework to generate customized and adaptable directions • Measure impact of adaptation on users

11. Formative Studies • Indoor and outdoor wayfinding using a "Wizard of Oz" system • Wide variation in user preference and expectation of directions • Modality (image, audio, text) • Complexity • Timing • Attention demand for turn-by-turn directions problematic • Alternative: landmark-based directions ` Liu et al. ASSETS 2006, PervasiveHealth 2009

12. Landmark-Based Directions • Landmark-based directions • Encourage attention to world • Orient user • More natural • What are features of good landmarks? • 9 study participants • TBI, stroke, MS, Dev Dis. A.L. Liu, H. Hile, H. Kautz, G. Borriello, P.A. Brown, M. Harniss, K. Johnson, PervasiveHealth 2009

13. Landmark Selection • Leverage collections of geo-tagged photos • Heuristic: more popular photos more useful • Can augment / warp photos to make them more closely match user's perspective

14. Results • 180 total directions given, 150 correctly followed • Much variation between participants • Features of most successful landmark-based directions: • In perspective • Close • Path-aligned • Making turns in relation to landmark required more cognitive effort • Turn-based directions should be user-centric

15. Modeling Individual Variation • Studies showed how to create a wayfinding system that was usually better for most users • However: system was still far from optimal for many users • Can we design a system that adapts to the individual user?

16. Adaptation as Optimization • Approach: for each user, learn a function: • Generating directions = finding optimal solution to a sequential decision problem • Issue: scarcity of user-specific training data • Solution: use data from all users, but weigh particular user's data more heavily

17. Adaptation Study • 5 participants • Compared performance on • Route generated with generic model • Route generated with personalized model • Qualitative result: personalized routes considered easier • Need larger, longer study to quantify significance

18. Summary • Studied wayfinding with real potential users • Learned range of abilities and preferences • Informed technological requirements • Incorporated landmarks as wayfinding option • Explored landmark dimensions that affect usability • Proof-of-concept of automatic adaptation • Creating a personal model using data from both the individual and the larger user base

19. Outline • Designing an adaptive wayfinding system for cognitively impaired users • Creating a context-aware task prompting system • Attention Control Systems: Richard Levinson • University of Rochester: Henry Kautz • VA Palo Alto: Dr. Harriet Zeiner • UC Santa Cruz: Beth Ann Hockey • Challenges in research on assisted cognition systems

21. PEAT • PEAT: handheld-based scheduling and cueing system for persons with executive function impairment (Attention Control Systems) • Multimedia prompts to start and end tasks • Assists with task (re)scheduling, by reasoning about deadlines, durations, and priorities

22. Problems with Prompting Systems • Require input from user whenever a task begins or ends • Cost of use may discourage users • Prompts may be out of sync • Self-initiation confuses system • Encourages dependency • No support for contingent prompts • Remind the user to take her walker when she goes outside

23. Integrating Sensing and Prompting • Goal: use information from sensors to improve prompting • Determine user's current activity from • Location: Home automation motion sensors, GPS • Objects user is handling: Intel Research Seattle RFID reader bracelet • Schedule of expected activities

24. Home sensors Cellphone RF Wireless connectionsGPS WiFi Relay WiFi State Estimator ZigBee PEAT RFID reader bracelet State DB Daily Activity Planning & Cueing PEAT User Interface User Context-Aware PEAT • R&D funded by grants from DARPA, Office of Defense, and NIH • Platform: Android cell phone • Internal sensors: GPS, acceleration • Home base station (laptop) for setup and relaying data from Insteon home automation sensors and Intel Research RFID reader • State estimator: employs probabilistic algorithms to infer user activity and learn user model

25. Reduced Prompting • Prompting cancelled if user self-initiates (or self-terminates) task • Reduces confusion / out of sync prompts • Encourages, rather than punishes, self-initiation • Can system do even more to encourage self-initiation? • Perhaps: prompt later than scheduled time • If prompt is too late and user does not self-initiate, task will fail • What is the optimal time for system to prompt?

26. Optimal Prompting • System can be configured to use a fixed prompting schedule (e.g. 5 minutes before scheduled time) or an adaptive schedule • Caregiver specifies rewards associated with • Successful (timely) task completion • User self-initiation of task • System • Learns model of user self-initiation • Computes prompting time that maximizes expected reward

27. Learning the User Model • User model includes probability distribution over how long it takes user to self-initiate • Technical challenge: how to estimate this parameter quickly and accurately? • Can use statistical techniques developed for reliability analysis

28. System Trial • In-home trials by Palo Alto VA start next month • Help regularize patients' daily schedule for • Sleep • Meals • Exercise • Behavioral therapy "homework"

29. Outline • Designing an adaptive wayfinding system for cognitively impaired users • Creating a context-aware task prompting system • Challenges in research on assisted cognition systems

30. Point versus General Solutions • “[A] tension exists between satisfying the need for customized solutions that address the immediate needs of users, and identifying generalizable results that can be used to achieve long-term research goals.” [Generalizability in Research with Cognitively Impaired Individuals, Moffatt, Findlater, Allen, 2006]

31. Evaluating with a Universe of One • “Twenty-eight papers presenting 25 studies were reviewed. The total number of participants was 423. Most identified papers described case reports or non-randomized clinical trials. Only one randomized controlled trial was identified.” [Efficacy and usability of assistive technology for patients with cognitive deficits: a systematic review, Joode, Heugten, Verhey, & Boxtel, 2010]

32. The Training Hurdle • “The [studies] that do describe the exact time spent on training show a substantial amount of variation (from 30 minutes up to 9 hours).” [Efficacy and usability of assistive technology for patients with cognitive deficits: a systematic review, Joode, Heugten, Verhey, & Boxtel, 2010]

33. Promise of Context-Aware Systems Medication Prompting: • “The context-based prompting resulted in significantly better adherence (92.3%) as compared to time-based (73.5%) or no prompting (68.1%) conditions (p < 0.0002, χ2 = 17.0). In addition, subjects had better adherence in the morning than in the evening. We have shown in this study that a system that generates reminders at an opportune time to take the medication significantly improves adherence.” [A Study of Medication-Taking and Unobtrusive, Intelligent Reminding, Hayes, Cobbinah, Dishongh, Kaye, Kimel, Labhard, Leen, Lundell, Ozertem, Pavel, Philipose, Rhodes, & Vurgun, 2009]

34. Pitfalls of Context-Aware Systems • “The participant was building a mental model of how the system behaved, and his model did not map well onto the behavior of the system. The problem was especially pronounced when an activity was not detected properly. ... The participant further admitted to changing his behavior in order to ‘fool‘ the system, but since he didn’t understand how it worked, this behavior was counter-productive. [User-adaptive Reminders for Home-based Medical Tasks, Kaushik, Intille, Larson, 2008]