1 / 61

Understanding Human Behavior from Sensor Data

Understanding Human Behavior from Sensor Data. Henry Kautz University of Washington. A Dream of AI. Systems that can understand ordinary human experience Work in KR, NLP, vision, IUI, planning… Plan recognition Behavior recognition Activity tracking Goals Intelligent user interfaces

angus
Download Presentation

Understanding Human Behavior from Sensor Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Understanding Human Behavior from Sensor Data Henry Kautz University of Washington

  2. A Dream of AI • Systems that can understand ordinary human experience • Work in KR, NLP, vision, IUI, planning… • Plan recognition • Behavior recognition • Activity tracking • Goals • Intelligent user interfaces • Step toward true AI

  3. Plan Recognition, circa 1985

  4. Behavior Recognition, circa 2005

  5. Punch Line • Resurgence of work in behavior understanding, fueled by • Advances in probabilistic inference • Graphical models • Scalable inference • KR U Bayes • Ubiquitous sensing devices • RFID, GPS, motes, … • Ground recognition in sensor data • Healthcare applications • Aging boomers – fastest growing demographic

  6. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

  7. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

  8. Object-Based Activity Recognition • Activities of daily living involve the manipulation of many physical objects • Kitchen: stove, pans, dishes, … • Bathroom: toothbrush, shampoo, towel, … • Bedroom: linen, dresser, clock, clothing, … • We can recognize activities from a time-sequence of object touches

  9. Application • ADL (Activity of Daily Living) monitoring for the disabled and/or elderly • Changes in routine often precursor to illness, accidents • Human monitoring intrusive & inaccurate Image Courtesy Intel Research

  10. Sensing Object Manipulation • RFID: Radio-frequency identification tags • Small • No batteries • Durable • Cheap • Easy to tag objects • Near future: use products’ own tags

  11. Wearable RFID Readers • 13.56MHz reader, radio, power supply, antenna • 12 inch range, 12-150 hour lifetime • Objects tagged on grasping surfaces

  12. Experiment: ADL Form Filling • Tagged real home with 108 tags • 14 subjects each performed 12 of 65 activities (14 ADLs) in arbitrary order • Used glove-based reader • Given trace, recreate activities

  13. Legend General solution Point solution Results: Detecting ADLs Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004

  14. Experiment: Morning Activities • 10 days of data from the morning routine in an experimenter’s home • 61 tagged objects • 11 activities • Often interleaved and interrupted • Many shared objects

  15. Baseline: Individual Hidden Markov Models 68% accuracy11.8 errors per episode

  16. Baseline: Single Hidden Markov Model 83% accuracy9.4 errors per episode

  17. Cause of Errors • Observations were types of objects • Spoon, plate, fork … • Typical errors: confusion between activities • Using one object repeatedly • Using different objects of same type • Critical distinction in many ADL’s • Eating versus setting table • Dressing versus putting away laundry

  18. Aggregate Features • HMM with individual object observations fails • No generalization! • Solution: add aggregation features • Number of objects of each type used • Requires history of current activity performance • DBN encoding avoids explosion of HMM

  19. Dynamic Bayes Net with Aggregate Features 88% accuracy6.5 errors per episode

  20. Improving Robustness • DBN fails if novel objects are used • Solution: smooth parameters over abstraction hierarchy of object types

  21. Abstraction Smoothing • Methodology: • Train on 10 days data • Test where one activity substitutes one object • Change in error rate: • Without smoothing: 26% increase • With smoothing: 1% increase

  22. Summary • Activities of daily living can be robustly tracked using RFID data • Simple, direct sensors can often replace (or augment) general machine vision • Accurate probabilistic inference requires sequencing, aggregation, and abstraction • Works for essentially all ADLs defined in healthcare literature D. Patterson, H. Kautz, & D. Fox, ISWC 2005 best paper award

  23. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

  24. Motivation: Community Access for the Cognitively Disabled

  25. The Problem • Using public transit cognitively challenging • Learning bus routes and numbers • Transfers • Recovering from mistakes • Point to point shuttle service impractical • Slow • Expensive • Current GPS units hard to use • Require extensive user input • Error-prone near buildings, inside buses • No help with transfers, timing

  26. Solution: Activity Compass • User carries smart cell phone • System infers transportation mode • GPS – position, velocity • Mass transit information • Over time system learns user model • Important places • Common transportation plans • Mismatches = possible mistakes • System provides proactive help

  27. Technical Challenge • Given a data stream from a GPS unit... • Infer the user’s mode of transportation, and places where the mode changes • Foot, car, bus, bike, … • Bus stops, parking lots, enter buildings, … • Learn the user’s daily pattern of movement • Predict the user’s future actions • Detect user errors

  28. Technical Approach • Map: Directed graph • Location: (Edge, Distance) • Geographic features, e.g. bus stops • Movement: (Mode, Velocity) • Probability of turning at each intersection • Probability of changing mode at each location • Tracking (filtering): Given some prior estimate, • Update position & mode according to motion model • Correct according to next GPS reading

  29. Dynamic Bayesian Network I mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k Time k-1

  30. Mode & Location Tracking Measurements Projections Bus mode Car mode Foot mode Green Red Blue

  31. Learning • Prior knowledge – general constraints on transportation use • Vehicle speed range • Bus stops • Learning – specialize model to particular user • 30 days GPS readings • Unlabeled data • Learn edge transition parameters using expectation-maximization (EM)

  32. Predictive Accuracy How to improve predictive power? Probability of correctly predicting the future City Blocks

  33. Transportation Routines Home A B Workplace • Goal: intended destination • Workplace, home, friends, restaurants, … • Trip segments: <start, end, mode> • Home to Bus stop A on Foot • Bus stopAto Bus stopB on Bus • Bus stop B to workplaceon Foot

  34. Dynamic Bayesian Net II gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k Time k-1

  35. Predict Goal and Path Predicted goal Predicted path

  36. Improvement in Predictive Accuracy

  37. Detecting User Errors • Learned model represents typical correct behavior • Model is a poor fit to user errors • We can use this fact to detect errors! • Cognitive Mode • Normal: model functions as before • Error: switch in prior (untrained) parameters for mode and edge transition

  38. ck-1 ck gk-1 gk tk-1 tk mk-1 mk xk-1 xk qk-1 qk zk-1 zk Time k Time k-1 Dynamic Bayesian Net III Cognitive mode { normal, error } Goal Trip segment Transportation mode Edge, velocity, position Data (edge) association GPS reading

  39. Detecting User Errors

  40. Status • Major funding by NIDRR • National Institute of Disability & Rehabilitation Research • Partnership with UW Dept of Rehabilitation Medicine & Center for Technology and Disability Studies • Extension to indoor navigation • Hospitals, nursing homes, assisted care communities • Wi-Fi localization • Multi-modal interface studies • Speech, graphics, text • Adaptive guidance strategies

  41. Papers Liu et al, ASSETS 2006 Indoor Wayfinding: Developing a Functional Interface for Individuals with Cognitive Impairments Liao, Fox, & Kautz, AAAI 2004 (Best Paper) Learning and Inferring Transportation Routines Patterson et al, UBICOMP 2004 Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services

  42. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

  43. Task • Learn to label a person’s • Daily activities • working, visiting friends, traveling, … • Significant places • work place, friend’s house, usual bus stop, … • Given • Training set of labeled examples • Wearable sensor data stream • GPS, acceleration, ambient noise level, …

  44. FRIEND’S HOME RESTAURANT OFFICE HOME PARKING LOT STORE Learning to Label Places & Activities • Learned general rules for labeling “meaningful places” based on time of day, type of neighborhood, speed of movement, places visited before and after, etc.

  45. Relational Markov Network • First-order version of conditional random field • Features defined by feature templates • All instances of a template have same weight • Examples: • Time of day an activity occurs • Place an activity occurs • Number of places labeled “Home” • Distance between adjacent user locations • Distance between GPS reading & nearest street

  46. RMN Model s1 Global soft constraints p1 p2 Significant places a1 a2 a3 a4 a5 Activities g1 g2 g3 g4 g5 g6 g7 g8 g9 {GPS, location} time 

  47. Significant Places • Previous work decoupled identifying significant places from rest of inference • Simple temporal threshold • Misses places with brief activities • RMN model integrates • Identifying significant place • Labeling significant places • Labeling activities

  48. Results: Finding Significant Places

  49. Results: Efficiency

More Related