Recognizing Activities of Daily Living from Sensor Data

1. Recognizing Activities of Daily Livingfrom Sensor Data Henry Kautz Department of Computer Science University of Rochester

2. Activity Recognition • Much recent interest in recognizing human activity from heterogeneous sensor data • Motion sensors • GPS • RFID • Video • Compelling applications • Military / security operations (e.g. ASSIST) • Smart homes & offices

3. Gathering data on indoor activities

4. Interpreting RFID Data (using Switching HMM)

5. Gathering Multi-view Video

6. Interpreting Video Computing scene statistics Ai = activity Oi = object Si = scene statistic Di = object statistics Ri = RFID label (for training) Computing object statistics

7. Gathering data on outdoor activities • Raw GPS

8. Discovering significant places Conditional Random Field

9. Predicting transportation goals Dynamic Bayesian Network

10. Issue • Previous work on activity recognition has used a wide variety of probabilistic models for different tasks and kinds of data • HMMs, DBNs, CRFs, … • Background knowledge is implicitly encoded in the structure of the models • E.g.: Relation between transportation goals, plans, actions • Increasingly difficult to scale & integrate

11. Markov Logic • Markov logic will provide common modeling language & inference tools, enabling • Easier integration of multiple sensors • Easier generalization • From one activity at a time to multiple ongoing activities • From one individual to multiple individuals • Easier modification of background knowledge • Add / modify library of plans and goals

12. Example Scenario • John goes into his kitchen (video) • He takes out a jug from the refrigerator, and a bowl from the cabinet (RFID) • He leaves his apartment, and walks to a convenience store (GPS) • He returns carrying a box (video) • He pours the box into the bowl (accelerometer) and the contents of the jug (accelerometer & RFID) • Why did John leave the apartment? What did he do?

13. UR Contributions to MURI: Scenario Development & Data Collection • Develop set of activity recognition scenarios of increasing complexity • Activities in the home • Outdoor activities • Enact and gather sensor data • Heterogeneous: GPS, RFID, video, motion, … • Intermittent and noisy • Make dataset available to team • Including feature sequences extracted from video and acceleration data • Ground truth • 1st data set mid-Year One, then ongoing

14. UR Contributions to MURI: Unified ML Model of Daily Activities • Recast our previous work on recognition using HMMs, DBNs, CRFs in Markov Logic • Integrate and generalize earlier results • Year One: • HMM  ML • Generalize to multiple ongoing activities • Handle novel observations using similarity • Representing actions, intentions, and goals • Extend ML to include “modal operators” • Distinguish beliefs of observer from beliefs of subject • Ability to model imperfect agents, whose plans are flawed

15. From HMMs to ML • Hidden Markov models describe the world as probabilistic state machine • ML encoding of HMM can be relaxed to allow subject to be in multiple states (multiple activities) by making “unique state” constraint soft

16. From HMMs to ML • Novel observations can be handled by applying background knowledge about similarity

17. Modal Operators • Most previous work on probabilistic activity recognition does not distinguish • What system infers is true about the world • What the subject believes is true about the world • What the system predicts will happen • What the subject intends to happen • Modal operators relate agents to attitudes • Bel( John, contains(jug, gasoline) ) • But system may know jug is empty • Goal( John, ignite(jug) ) • Knowledge of subject’s goal can drive cooperative system to help subject, or antagonistic system to block user

18. Semantic Inference • Modal operators do not work like ordinary predicates or logical connectives • Modal proof theory is hard to automate • However: • Modal operators have well-understood “possible world” semantics • Agent believes P in possible world W iff P is true in all worlds W’ such that reachable(W,W’) • ML’s inference engine works at the semantic level (direct search over possible worlds) • Promising approach: semantic inference for modal constructs in ML • Explicitly model reachability relationships for each attitude and agent

19. Idea • Alchemy searches over models (truth assignments) • Modal formulas are evaluated over structures • Structure = set of models and accessibility relationships over the models • Structures are too big to explicitly search • Modify Alchemy to search over samples drawn from structures