1 / 62

Enhancing Independent Living With Wearable Activity Recognition Tech

Develop technology supporting independent living for the aging population & Alzheimer's patients. Monitor activities, intervene before crises, improve healthcare through wearable sensors & AI. Transportation routines, significant places, & custom models personalized for each user. Technology funded by NSF, DARPA, MSR, Toyota for long-term monitoring in homes, workplaces, & communities. Activity Recognition using Dynamic Bayes Nets & Conditional Random Fields to predict, detect novel behaviors & user errors for better healthcare outcomes.

irmam
Download Presentation

Enhancing Independent Living With Wearable Activity Recognition Tech

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dieter Fox University of Washington Department of Computer Science & Engineering

  2.  Joint work with Students: Brian Ferris, Julie Letchner, Lin Liao, Don Patterson, Alvin Raj, Amarnag Subramanya, Danny Wyatt UW faculty: Jeff Bilmes, Gaetano Borriello, Henry Kautz Intel Research Seattle: Tanzeem Choudhury, Matthai Philipose  Funded by NSF, DARPA, MSR, Toyota 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 2

  3.  U.S. by 2050: 80 million people over age 65 and 13-16 million Alzheimer's patients  Goal: Develop technology to Support independent living by people with cognitive disabilities ▪ At home ▪ At work ▪ Throughout community Improve health care ▪ Long term monitoring of activities of daily living (ADL’s) ▪ Intervention before a health crisis 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 3

  4.  Transportation routines via Dynamic Bayes Nets  Significant places via Conditional Random Fields  Activity recognition projects  Discussion 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 4

  5. [Liao-Fox-Kautz: AAAI-04, AIJ-07]  Given data stream from a wearable GPS unit Infer the user’s location and mode of transportation (foot, car, bus, bike, ...) Predict where user will go Detect novel behavior / user errors 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 5

  6.  Dead and semi-dead zones near buildings, trees, etc.  Sparse measurements inside vehicles, especially bus  Multi-path propagation  Inaccurate street map  … 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 6

  7.  Map is directed graph  Location: Edge e Distance d from start of edge  Prediction: Move along edges according to velocity model  Correction: Update estimate based on GPS reading 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 7

  8. xk-1 xk Edge, velocity, position zk-1 zk GPS reading Time k Time k-1 Task: Estimate posterior over hidden states 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 8

  9. e1 x k-1 e3 ? e2 e0 ? Problem: Predicted location is multi-modal 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 10

  10. e1 x k zk e3 e2 Problem: GPS reading is not on the graph 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 11

  11. x k e1 x if =e1 q k zk e3 e2 Problem: GPS reading is not on the graph 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 12

  12. e1 x k zk x k e3 if =e2 q e2 Problem: GPS reading is not on the graph 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 13

  13. ek-1 ek Edge transition xk-1 xk Edge, velocity, position GPS association θ k θ k-1 zk-1 zk GPS reading Time k Time k-1 Task: Estimate posterior over all hidden states 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 14

  14. STREET MAP Source: Tiger / Line data BUS ROUTES / STOPS Source: Metro GIS RESTAURANTS / STORES Source: MS MapPoint 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 15

  15.  Represents posterior by sets of weighted particles: , { s S k   ( ) ( ) i i , ,..., 1  } w i n   Basic idea: Sample some variables of state space and solve other variables analytically conditioned on samples  Here: Each particle contains Kalman filter for location: , ,  v e s    ( ) ( ) ( ) ( ) ( ) 2 i i i i i , ( , ) N KF for position Edge transitions, velocities, edge associations  Update sets by sampling procedure (SISR) 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 16

  16. GPS measurements Particles (Kalmanfilters) 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 17

  17. mk-1 mk Transportation mode xk-1 xk Edge, velocity, position zk-1 zk GPS reading Time k Time k-1  ( ,   ( ) i ( ) i ( ) i ( ) i ( ) i ( ) i 2 , , , , ) s m e v N  Particles: 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 18

  18.  Encode prior knowledge into the model Modes have different velocity distributions Buses run on bus routes Get on/off the bus near bus stops Switch to car near car location 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 19

  19. Measurements Projections Bus mode Car mode Foot mode Green Red Blue 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 20

  20. A B Workplace  Goal (destination): workplace (home, friends, restaurant, ...)  Trip segments: <start, end, transportation> Home to Bus stop A on Foot Bus stop Ato Bus stop B on Bus Bus stop B to workplace on Foot 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 21

  21. [Liao-Fox-Kautz: AAAI-04, AIJ-07] gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position zk-1 zk GPS reading Time k Time k-1 [Bui-Venkatesh-West: JAIR-02] ( ) i  ( ,   ( ) i ( ) i ( ) i ( ) i ( ) ( ) i 2 i , , , , , , ) s g t m e v N  Particles: 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 22

  22.  Customized model for each user  Key to goal / path prediction and error detection  Use low level model to learn variable domains Goals: locations where user stays for long time Transition points: locations with high mode transition probability Trip segments: connecting transition points or goals  Use full model to learn parameters Transition probabilities (goals, trips, edges) via EM 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 23

  23. Predicted goal Predicted path 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 24

  24. GOING TO THE WORKPLACE GOING HOME High probability transitions: bus car foot 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 25

  25. 0.5 0.5 5 43 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 26

  26. [Patterson-Liao-etAl: Ubicomp-04] bk-1 bk Behavior mode normal / unknown / error gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position zk-1 zk GPS reading Time k Time k-1 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 27

  27. 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 28

  28. Untrained flat model Trained model Instantiated model 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 29

  29. 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 30

  30.  Transportation routines via Dynamic Bayes Nets  Significant places via Conditional Random Fields  Activity recognition projects  Discussion 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 31

  31. [Liao-Fox-Kautz: NIPS-05,IJCAI-05,IJRR-07]  So far No distinction between different types of goals Model not transferable to other person Goals learned solely based on duration 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 32

  32. Friend Restaurant Work Home Bus stop Parking Bus stop Store 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 33

  33.  Undirected graphical model  Introduced for labeling sequence data [Lafferty-et al: ICML-01]  No independence assumption on observations!  Trained discriminatively from labeled data  Applied successfully to Natural language processing [McCallum-Li: CoNLL-03], [Roth-Yih: ICML-05] Computer vision [Kumar-Hebert: NIPS-04], [Quattoni-Collins-Darrel: NIPS-05] Robotics [Limketkai-Liao-Fox: IJCAI-05], [Friedman-Pasula-Fox: IJCAI-07] 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 34

  34. ConditionalRandomFields x Hidden states x k 1  Observations z  Posterior factorizes into log-linear clique potentials 1 ( | ) ( , ( ) c C Z   z 1 ( ) z       T c c C   x z x z w x z ) exp f ( c , ) p    c c c c c Z  Local potentials link states to observations / features  Neighborhood potentials link states to neighboring states 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 35

  35. ConditionalRandomFields x Hidden states x k 1  Observations z  Posterior factorizes into log-linear clique potentials 1 ( | ) ( , ( ) c C Z   z 1 ( ) z       T c c C   x z x z w x z ) exp f ( c , ) p    c c c c c Z  Local potentials link states to observations / features  Neighborhood potentials link states to neighboring states 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 36

  36. ConditionalRandomFields x Hidden states x k 1  Observations z  Posterior factorizes into log-linear clique potentials 1 ( | ) ( , ( ) c C Z   z 1 ( ) z       T c c C   x z x z w x z ) exp f ( c , ) p    c c c c c Z Partition function Weights Feature functions  Local potentials link states to observations / features  Neighborhood potentials link states to neighboring states 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 37

  37.  Conditional distribution parameterized via weights w: 1 ( | , ) exp ( , ) c C Z   z w      T c x z w w x z f ( c , ) p  c c   Maximize conditional log-likelihood with shrinkage prior: T log ( | , ( ) ) L p  w w w x z w   2  Maximization via stochastic / conjugate gradient, L-BFGS  Alternative: maximize pseudo log-likelihood [Besag: 1975] T w w log (  w x x w PL( ) |MB( ), ) p   i i  2 i 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 38

  38.  BP computes posteriors via local message passing Sum-product for posterior .... Max-product for MAP ....  Exact if network has no loops  Otherwise, run loopy belief propagation and hope it works  Alternatives: ICM, graph-cut, MCMC, … 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 39

  39.  Temporal pattern: duration, time of day, etc.  Geographic evidence: is there a restaurant / store / bus stop nearby?  Transition relation: adjacent activities  Spatial feature: relation between place and activity  Summation constraint: number of places labeled home or workplace 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 40

  40. [Liao-Fox-Kautz: NIPS-05] Activity sequence walk, drive, ride bus, work, visit, sleep, pickup, get on/off bus … a1 a2 a3 a4 a5 aN-2 aN-1 aN … GPS trace 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 41

  41. [Liao-Fox-Kautz: NIPS-05] Global, soft constraints # homes, workplaces h w Significant places home, work, bus stop, parking lot, friend’s home … p1 p2 p3 pK Activity sequence walk, drive, ride bus, work, visit, sleep, pickup, get on/off bus … a1 a2 a3 a4 a5 aN-2 aN-1 aN … GPS trace  Two weeks data (> 50,000 gps points)  Inference: loopy BP (10 minutes); Learning: Pseudo-likelihood (3 minutes) 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 42

  42.  Generalize from people with labeled data to others without labeled data User1 Supervised learning Training data Generic model Other users User2 Shared parameters User3  Relational Markov Networks to specify CRFs and parameter sharing via templates [Taskar-Abbeel-Koller: UAI-02] 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 43

  43. 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 44

  44. 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 45

  45. 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 46

  46. Cross-validation using data from 4 persons 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 47

  47. 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 48

  48.  Transportation routines via Dynamic Bayes Nets  Significant places via Conditional Random Fields  Activity recognition projects  Discussion 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 49

  49. [Ferris-Haehnel-Fox: RSS-06] 573 AU-08 Dieter Fox: Activity Recognition From Wearable Sensors 50

More Related