Robust Activity Recognition

1. Robust Activity Recognition Henry Kautz University of WashingtonComputer Science & Engineering graduate students: Don Patterson, Lin Liao,Krzysztof Gajos, Karthik Gopalratnam CSE faculty: Dieter Fox, Gaetano Borriello UW School of Medicine: Kurt Johnson, Pat Brown, Brian Dudgeon, Mark Harniss Intel Research: Matthai Philipose, Mike Perkowitz, Ken Fishkin, Tanzeem Choudhury

2. In the Not Too Distant Future... • Pervasive sensing infrastructure • GPS enabled phones • RFID tags on all consumer products • Electronic diaries (MS SenseCam) • Healthcare crisis • Aging baby boomers – epidemic of Alzheimer’s Disease • Deinstitutionalization of the cognitively disabled • Nationwide shortage of caretaking professionals

3. ...An Opportunity • Develop technology to • Support independent living by people with cognitive disabilities • At home • At work • Throughout the community • Improve health care • Long term monitoring of activities of daily living (ADL’s) • Intervention before a health crisis

4. The UW Assisted Cognition Project • Synthesis of work in • Ubiquitous computing • Artificial intelligence • Human-computer interaction • ACCESS • Support use of public transit • UW CSE & Rehabilitation Medicine • CARE • ADL monitoring and assistance • UW CSE & Intel Research

5. This Talk • Building models of everyday plans and goals • From sensor data • By mining textual description • By engineering commonsense knowledge • Tracking and predicting a user’s behavior • Noisy and incomplete sensor data • Recognizing user errors • First steps

6. ACCESSAssisted Cognition in Community, Employment, & Support SettingsSupported by the National Institute on Disability & Rehabilitation Research (NIDDR) Learning & Reasoning About Transportation Routines

7. Task • Given a data stream from a wearable GPS unit... • Infer the user’s location and mode of transportation (foot, car, bus, bike, ...) • Predict where user will go • Detect novel behavior • User errors? • Opportunities for learning?

8. Why Inference Is Not Trivial • People don’t have wheels • Systematic GPS error • We are not in the woods • Dead and semi-dead zones • Lots of multi-path propagation • Inside of vehicles • Inside of buildings • Not just location tracking • Mode, Prediction, Novelty

9. GPS Receivers We Used GeoStats wearable GPS logger Nokia 6600 Java Cell Phone with Bluetooth GPS unit

10. Geographic Information Systems Street map Data source: Census 2000 Tiger/line data Bus routes and bus stops Data source: Metro GIS

11. Architecture Learning Engine • Goals • Paths • Modes • Errors GIS Database Inference Engine

12. Probabilistic Reasoning • Graphical model: Dynamic Bayesian network • Inference engine: Rao-Blackwellised particle filters • Learning engine: Expectation-Maximization (EM) algorithm

13. Flat Model: State Space • Transportation Mode • Velocity • Location • Block • Position along block • At bus stop, parking lot, ...? • GPS Offset Error • GPS signal

16. Motion Model for Mode of Transportation

17. Rao-Blackwellised Particle Filtering • Inference: estimate current state distribution given all past readings • Particle filtering • Evolve approximation to state distribution using samples (particles) • Supports multi-modal distributions • Supports discrete variables (e.g.: mode) • Rao-Blackwellisation • Particles include distributions over variables, not just single samples • Improved accuracy with fewer particles

18. Tracking blue = foot, green = bus, red = car

19. Learning • User model = DBN parameters • Transitions between blocks • Transitions between modes • Learning: Monte-Carlo EM • Unlabeled data • 30 days of one user, logged at 2 second intervals (when outdoors) • 3-fold cross validation

20. Results

21. Prediction Accuracy How can we improve predictive power? Probability of correctly predicting the future City Blocks

22. Transportation Routines A B Work • Goals • work, home, friends, restaurant, doctor’s, ... • Trip segments • Home to Bus stop A on Foot • Bus stop A to Bus stop B on Bus • Bus stop B to workplace on Foot

23. Hierarchical Model gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk x=<Location, Velocity> zk-1 zk GPS reading

24. Hierarchical Learning • Learn flat model • Infer goals • Locations where user is often motionless • Infer trip segment begin / end points • Locations with high mode transition probability • Infer trips segments • High-probability single-mode block transition sequences between segment begin / end points • Perform hierarchical EM learning

25. Inferring Goals

26. Inferring Trip Segments Going to work Going home

27. Correct goal and route predicted 100 blocks away

28. Application:Opportunity Knocks Demonstrated at AAHA Future of Aging Services, Washington, DC, March, 2004

29. Novelty Detection • Approach: model-selection • Run two trackers in parallel • Tracker 1: learned hierarchical model • Tracker 2: untrained flat model • Estimate the likelihood of each tracker given the observations

30. Missing the bus stop

31. Novelty Detection

33. CARECognitive Assistance in Real-world Environmentssupported by the Intel Research Council Learning & Inferring Activities of Daily Living

34. Research Hypothesis • Observation: activities of daily living involve the manipulation of many physical objects • Cooking, cleaning, eating, personal hygiene, exercise, hobbies, ... • Hypothesis: can recognize activities from a time-sequence of object “touches” • Such models are robust and easily learned or engineered

35. Sensing Object Manipulation • RFID: Radio-frequency ID tags • Small • Semi-passive • Durable • Cheap

36. Where Can We Put Tags?

37. How Can We Sense Them? coming... wall-mounted “sparkle reader”

38. Example Data Stream

39. Technical Approach • Define (or learn) activities in simple, high-level language • Multi-step, partially-ordered activities • Varying durations • Probabilistic association between activities and objects • Compile to a DBN • Infer behavior using particle filtering

40. Making Tea

41. Activity Library

42. Building Models • Core ADL’s amenable to classic knowledge engineering • Open-ended, fine-grained models: infer from natural language texts? • Perkowitz et al., “Mining Models of Human Activities from the Web”, WWW-2004

43. Translation to DBN • Tricky issues: • Time • Partial orders • Object-use probabilities • 80% chance of using the teapot sometime during the “heat water” step • Instantaneous probability of seeing teapot is not fixed! • Consider: 100% chance of using teapot if making tea

44. DBN Encoding: Duration At At+1 Dt Dt+1

45. DBN Encoding: Partial Orders Pt Pt At At+1

46. DBN Encoding: Object Probabilities At Instantaneous probability of touching an object cannot be a constant Dt Ht Ot zt

47. DBN Encoding Pt Pt At At+1 Dt Dt+1 Ht Ht+1 Ot zt

48. What’s in a Particle? • Sample of Activity • Starting time – sufficient to represent distribution of Duration • History list of objects • Partial-order “credits”

49. Experimental Setup • Hand-built library of 14 ADL’s • 17 test subjects • Each asked to perform 12 of the ADL’s • Data not segmented • No training on individual test subjects

50. Sample Output