1 / 36

The Assisted Cognition Project

The Assisted Cognition Project. Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine) Kurt Johnson, Pat Brown, Mark Harniss (UW Rehabilitation Medicine) Matthai Philipose (Intel Research Seattle).

orien
Download Presentation

The Assisted Cognition Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano BorielloLin Liao, Brian Ferris, Evan Welborne(UW CSE) Don Patterson(UW / UC Irvine) Kurt Johnson, Pat Brown, Mark Harniss(UW Rehabilitation Medicine) Matthai Philipose(Intel Research Seattle)

  2. Trend 1: Sensing Infrastructure • Robust direct-sensing technology • GPS-enabled phones • RFID tagged products • Wearable multi-modal sensors • Rapid commercial deployment

  3. Trend 2: Healthcare Crisis • Demand for community integration of the cognitively disabled • 100,000 @ year disabled by traumatic brain injury • 7.5 million in US with mental retardation • 4 million in US with Alzheimer’s • Family burnout • Nationwide shortage of professionals

  4. Assisted Cognition • Technology to support independent living by people with cognitive disabilities • at home • at work • throughout the community by • Understanding human behavior from sensor data • Actively prompting and advising • Alerting human caregivers when necessary

  5. Building Partnerships • UW Assisted Cognition seminar • CSE, medicine, nursing, Intel • ACCESS • UW CSE & Rehabilitation Medicine • Grant from NIDDR (Dept. of Education) • Help cognitively disabled use public transportation • Prototype: Opportunity Knocks • Intel Proactive Health effort • Computing for wellness & caregiving • Promote partnerships with government, universities, healthcare organizations • Intel Seattle: sensors for activity tracking

  6. Example • Way-finding Assistant • Help user travel throughout community • On foot • Using public transportation • Detect user errors • Proactively help user recover • “You missed your stop, so get off at the next stop and then wait for the #16 bus...” • Potential users • TBI, MR, mild memory impairment

  7. Example • ADL Assistant • Activities of daily living • Eating, bathing, dressing, ... • Cooking, cleaning, emailing, ... • Monitoring • Changes in ADLs signal changes in health • Reminding / prompting • “Time to take your blue meds” • Step-by-step guidance • “Turn on the tap ... now pick up the brush ...” • Potential users • Disabled, ordinary aging

  8. General Model user model geospatial DB intervention decision making common- sense KB userinterface caregiveralerts wearablessensors environmentalsensors

  9. cognitive state goals activity General Model geospatial DB intervention decision making common- sense KB physical motion & position userinterface caregiveralerts wearablessensors environmentalsensors

  10. Deciding to Intervene A = system intervenes G = user actually needs help

  11. ACCESSWay-finding Assistant supported by National Institute on Disability & Rehabilitation Research DARPA IPTO

  12. The Need: Community Access for the Cognitively Disabled

  13. Problems in Using Public Transportation • Learning bus routes and numbers

  14. Problems in Using Public Transportation • Learning bus routes and numbers • Transfers, complex plans

  15. Problems in Using Public Transportation • Learning bus routes and numbers • Transfers, complex plans • Recovering from mistakes

  16. Result • Need for extensive life-coaching • Need for point-to-bus service

  17. Result • Need for extensive life-coaching • Need point-to-bus service • Isolation

  18. Current GPS Navigation Devices • Designed for drivers, not bus riders! • Should I get on this bus? • Is my stop next? • What do I do if I miss my stop? • Requires extensive user input • Keying in street addresses no fun! • Device decides which route is “best” • Familiar route better than shorter one • “Catastrophic failure” when signal is lost

  19. New Approach • User carries GPS cell phone • System infers transportation mode • Position, velocity, geographic information • Over time, system learns about user • Important places • Common transportation plans • Breaks from routine = possible user errors • Ask user if help is needed

  20. User Model ck-1 ck Cognitive mode { routine, novel, error } gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k Time k-1

  21. Error Detection: Missed Bus Stop

  22. Prototype: Opportunity Knocks • GPS camera-phone • “Knocks” when there is an opportunity to help • Can I guide you to a likely destination? • I think you made a mistake! • This place seems important – would you photograph it?

  23. Status • User needs study • Algorithms for learning and predicting transportation behavior • Best paper award at AAAI-2004 • Proof of concept prototype • Now: user interface studies • Modality: Audio, Graphics, Tactile, ... • Guidance strategies: Landmarks, User frame of reference, Maps, ...

  24. ADL Monitoring from RFID Tag Data UW CSE Intel Research Seattle demo at Intel this afternoon

  25. Object-Based Activity Recognition • Activities of daily living involve the manipulation of many physical objects • Kitchen: stove, pans, dishes, … • Bathroom: toothbrush, shampoo, towel, … • Bedroom: linen, dresser, clock, clothing, … • We can recognize activities from a time-sequence of object touches

  26. Sensing Object Manipulation • RFID: Radio-frequency identification tags • Small • Long-lived – no batteries • Durable • Easy to deploy • Bracelet touch sensor • Wall-mount movement sensor

  27. Example Data Stream

  28. Example Activity Model

  29. Creating Models of ADLs • Hand-built • Learn from sensor data • Mine from natural-language texts • All of the above...

  30. Experiment: Morning Activities • 10 days of data from the morning routine in an experimenter’s home • 61 tagged objects • 11 activities • Often interleaved and interrupted • Many shared objects

  31. DBN with Aggregate Features 88% accuracy6.5 errors per episode

  32. Improving Robustness • Tracking fails if novel objects are used • Solution: smooth parameters over abstraction hierarchy of object types

  33. Status • Accurate tracking of wide variety ADLs • Active collaboration with Intel • Current work • Detecting user errors in ADL performance • Learning more complex ADLs • Preconditions/effects • Multi-tasking • Temporal constraints • Reminding & prompting

  34. Concluding Remarks • Research on Assisted Cognition going great guns at UW and (a few) other universities • CMU / Pitt / U Michigan (Nursebot, Autominder – M. Pollack) • Georgia Tech (Aware Home, G. Abowd) • MIT (House N, Stephen Intille)

  35. Some Thoughts on Funding • Getting funding for work in this area is currently challenging • We were fortunate once with NIDRR, but less than 1% of their budget is for research • NIH & NIA spend relatively little on caregiving research • New NIH “Roadmap” for interdisciplinary exploratory research completely leaves out caregiving! • NIN has good people, but no real money

  36. Some Thoughts on Funding • Getting funding for work in this area is currently challenging • NSF supports some of the underlying, multi-use technology, but not medically-oriented applications • Exception: helping disabled use computers • Industry support is vital, but more for collaboration than actual dollars • Good industry grant = 1 grad student • There’s a gap waiting to be filled...

More Related