Adaptive Social Assistants: Using Mobile Computing Devices to Assist Individuals with Cognitive Disabilities

1. Adaptive Social Assistants: Using Mobile Computing Devices to Assist Individuals with Cognitive Disabilities Project Update Cathy Bodine University of Colorado June 14, 2001

2. Project Objectives • Develop power-aware mobile computing devices that adapt to their users based upon observed and predicted behavior. • Configure these devices as adaptive social assistants that simplify daily living and job-related tasks for persons with cognitive disabilities. • The role of the social assistant is to replace many of the functions of a personal attendant, while enhancing the independence and functioning of the user as he or she engages in daily life activities as a contributing member of society.

3. Project Team • John Bennett- CU Dept. of Comp. Science • Dirk Grunwald - CU Dept. of Comp. Science • Clayton Lewis - CU Dept. of Comp. Science • Michael Mozer - CU Dept. of Comp. Science • Tamara Sumner - CU Dept. of Comp. Science • Timothy Brown - CU Dept. of ECE and ITP • Cathy Bodine- UCHSC Depts. Of Peds. & Rehab. Med.; Director, Assistive Technology Partners • Linda Crnic -UCHSC Depts. of Pediatrics and Psychiatry • Deborah Fidler -CSU Dept. of Human Development and Family Studies • Lori Ramig -CU Dept. ofSpeech, Lang., and Hearing Sci. • Sally Rogers -UCHSC Dept. of Psychiatry • David Patterson -UCHSC Depts. of Biochemistry and Molecular Genetics and Medicine

4. Project Expertise by Area • John Bennett (distributed, parallel, and mobile computing) • Dirk Grunwald (computer systems and networking) • Clayton Lewis (human-computer interaction) • Michael Mozer (machine learning) • Tamara Sumner (design and cognition, HCI) • Timothy Brown (wireless computing) • Cathy Bodine(assistive technology) • Linda Crnic (mental retardation) • Deborah Fidler (developmental disabilities, esp. outcomes) • Lori Ramig (speech pathology and remediation) • Sally Rogers (developmental disabilities, esp. autism) • David Patterson (genetics of developmental disorders)

5. Project Status • Preproposal ($5M budget) submitted Fall ’00 (1 of 661 submitted) • Favorable action on preproposal by NSF (1 of 258 (39%) approved for full proposal) • Full proposal submitted April ’01 (Sept. ’01 start date if funded) (about $90M allocated => ~ 50 proposals will be funded) • $1M matching support pledged from Coleman Institute

6. Core Technology • Machine Learning (the means by which the social assistants learn from and adapt to user behavior) • Mobility and Data Management (power management, how we support both user and device mobility, and how we represent, access, update, and protect information) • Human-Computer Interaction (how the user interacts with the assistant)

7. Target Tasks • Route Navigation • Assistant provides auditory / verbal instructions and observes trajectories to build up expectations. Significant violations of these expectations results in warning or other remedial action. • Communication • Assistant provides augmentative / alternative communication. “Keypad” adapts to user (not the other way around). We intend to build on work of Enkidu and Saltillo. • Memory Prosthesis • Assistant provides memory cues to help user stay on and complete tasks, as well improving ability to complete tasks more successfully. We intend to build upon work of AbleLink, integrating adaptive behavior into “Pocket Coach” like devices.

8. Design Issues • Safety, Welfare & Privacy • Reliability, Durability, Ease of Use & Wearability • High Abandonment Rate of Assistive Devices • Training • The Role of Adaptation

9. Participant Selection • Working or job-seeking adult aged ~16 - ~40 • Diagnosed with developmental disability such as one of Down, Williams, fragile X, or Prader-Willi syndromes, or autism • Ambulatory and within gross normal visual and hearing acuity limits • Able to follow single step instructions • Must have 24/7 emergency support • For now, exclude those with clinical range aberrant behaviors, serious communicable diseases, severe motor impairments

10. Syndrome prevalence • Down syndrome (the most common genetic mental retardation syndrome) • 1 in 700-1000 live births • Williams syndrome • 1 in 20,000 live births • Prader-Willi syndrome • 1 in 10,000 live births • fragile X syndrome • 1 in 2,000- 4,000 live births

11. Targeting Syndrome Differences • Persons with Down Syndrome • Tend to require visual support of auditory information • Show strength in sequential over simultaneous processing => social assistant should provide step-by-step instructions • Persons with Williams Syndrome • Tend to require auditory support of visual information • Persons with Prader-Willi or fragile X syndrome • Show strength in simultaneous over sequential processing => social assistant should provide instructions in larger, integrated context

12. Broader Impact of Project • Increasing independence and improving quality of life of people with cognitive disabilities • Extension to general population, including: • the elderly • persons who cannot read • persons who are blind • persons with traumatic brain injury (impaired memory) • anyone who needs help with daily tasks, including busy executives and college professors • Education and outreach • Improving our understanding of individuals with cognitive disabilities • Technology transfer

13. Five Year Research Plan • Year 1 – [Fact Gathering, Exploration, and Participatory Design] • Year 2 – [Deliver First (Basic) Prototypes] • Year 3 – [Evaluation; Additional Deployment and Testing] • Year 4 – [Design Refinement] • Year 5 - [Evaluation, Analysis, and Technology Transfer]

14. Year 1 – [Fact Gathering, Exploration, and Participatory Design] • Establish dialogue with users, family members, caregivers, clinicians and manufacturers of assistive technology devices in order to develop an understanding of users needs and capabilities related to the target tasks. Refine target tasks as necessary. • Design and implement data encryption, transmission and transaction mechanisms. • Establish project web site. • Begin development of laboratory prototypes of social assistants. • Begin exploration of different user interface mechanisms and sensor technologies. • Develop appropriate machine learning objective functions for device adaptation. • Experiment with cellular, GPS, and other location sensing technologies. • Complete Institutional Review Board approval process prior to device deployment in Year 2. • Develop and offer joint course in assistive technology for engineering and health science graduate students. • Establish project Advisory Board

15. Year 2 – [Deliver First (Basic) Prototypes] • Complete high-level design of system architecture. • Establish initial server infrastructure for both development and field use. • Develop basic data and mobility management support mechanisms. • Develop initial task modeling interface for use by clinicians and caregivers. • Perform initial testing of machine learning objective functions for device adaptation. • Develop initial user interface design. • Deploy 40 field units (20 at midyear, 20 more by year end.) Release new software to field at 6-month intervals. • Begin to collect and analyze user data.

16. Year 3 – [Evaluation; Additional Deployment and Testing] • Expand user population and task repertoire. • Expand server infrastructure as needed. • Refine and extend task modeling interface for use by clinicians and caregivers. • Evaluate device adaptation. • Extend basic data and mobility management support mechanisms. • Complete initial user interface design. • Deploy an additional 40 field units (20 every 6 months). Release new software to field at 6-month intervals.

17. Year 4 – [Design Refinement] • Expand user population and task repertoire. • Expand server infrastructure as needed. • Develop final task modeling interface for use by clinicians and caregivers. • Continue to evaluate device adaptation and refine ML objective functions based on user data. • Evaluate and refine user interface design based upon field data. • Deploy an additional 40 field units (20 every 6 months). Release new software to field at 6-month intervals. • Initiate technology transfer.

18. Year 5 - [Evaluation, Analysis, and Technology Transfer] • Expand user population to full size and complete task repertoire. • Expand server infrastructure as needed. • Finalize task modeling interface for use by clinicians and caregivers. • Finalize user interface design. • Evaluate device adaptation based on analysis of user data. • Deploy 20 additional units and final software release at mid year. • Perform comprehensive evaluation of system architecture and design. • Complete transfer of project technology to commercial developer.