Raffay Hamid, Amos Johnson, Samir Batta, Aaron Bobick, Charles Isbell, Graham Coleman

1. Activity Discovery and Anomalous Activity Explanation Raffay Hamid, Amos Johnson, Samir Batta, Aaron Bobick, Charles Isbell, Graham Coleman

2. Activity Discovery and Anomalous Activity Explanation

3. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular”

4. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly

5. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly

6. Activity Representation • Previous representations include: • Stochastic Context Free Grammars • Expectation Grammars • …..

7. Activity Representation • Previous representations include: • Stochastic Context Free Grammars • Expectation Grammars • ….. • Require some a priori information about activity structure

8. Activity Representation • Two pieces of information: • content • structure • Drawing from Natural Language Processing – treating documents as bags of words • Treat Activities as bags of event n-grams • Extraction of global structural information using local event statistics

9. Activity Representation • Two pieces of information: • content • structure • Drawing from Natural Language Processing – treating documents as bags of words –captures content well • Treat Activities as bags of event n-grams • Extraction of global structural information using local event statistics

10. Activity Representation • Two pieces of information: • content • structure • Drawing from Natural Language Processing – treating documents as bags of words –captures content well • Treat Activities as bags of event n-grams –captures activity structure • Extraction of global structural information using local event statistics

11. Activity Representation • Two pieces of information: • content • structure • Drawing from Natural Language Processing – treating documents as bags of words –captures content well • Treat Activities as bags of event n-grams –captures activity structure • Extraction of global structural information using local event statistics

14. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly

15. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly - Occur frequently - Are similar to each other

16. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly - Activity similarity - Activity discovery

17. Activity Similarity • Two types of differences • core structural differences (csd) • event frequency differences (efd) • Sim (A,B) = w1*CSD(A,B) + w2*EFD(A,B) • Properties: • Identity • Commutative • Positive semi-definite

18. Activity Similarity • Two types of differences • core structural differences (csd) • event frequency differences (efd) • Properties: • Identity • Commutative • Positive semi-definite

19. Activity Similarity • Two types of differences • core structural differences (csd) • event frequency differences (efd) • Properties: • Identity • Commutative • Positive semi-definite

20. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly Activity similarity - Activity discovery

21. Activity Sub-Class Discovery • Recall: regular activities occur frequently and are similar to each other • Activity Sub-Class Discovery - a Graphic Theoretic problem of finding maximal cliques in edge-weighted graphs • Maximal Cliques: overall similarity between clique nodes greater than some value, addition of any other node would reduce the overall clique similarity

22. Activity Sub-Class Discovery • Recall: regular activities occur frequently and are similar to each other • Activity Sub-Class Discovery - a Graphic Theoretic problem of finding maximal cliques in edge-weighted graphs • Maximal Cliques: overall similarity between clique nodes greater than some value, addition of any other node would reduce the overall clique similarity

23. Activity Sub-Class Discovery • Recall: regular activities occur frequently and are similar to each other • Activity Sub-Class Discovery - a Graphic Theoretic problem of finding maximal cliques in edge-weighted graphs • Maximal Cliques: overall similarity between clique nodes greater than some value, addition of any other node would reduce the overall clique similarity

24. Activity Sub-Class Discovery • Sequentially find maximal cliques in edge weighted graph of activities • Activities different enough from all the regular activities are anomalies

25. Activity Sub-Class Discovery • Sequentially find maximal cliques in edge weighted graph of activities • Activities different enough from all the regular activities are anomalies

26. Activity Sub-Class Discovery • Sequentially find maximal cliques in edge weighted graph of activities • Activities different enough from all the regular activities are anomalies

27. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly

28. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly Activity classification - Anomaly detection

29. Activity Classification • Compute weighted similarity between a new activity T and previous class members as: • Select membership sub-class as:

30. Activity Classification • Compute weighted similarity between a new activity T and previous class members as: • Select membership sub-class as:

31. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly Activity classification - Anomaly detection

32. Anomaly Detection • Define function • Learn the detection threshold from training data

33. Anomaly Detection • Define function • Represents the within-Class difference of the test activity w.r.t. previous class members • Pick a particular threshold to detect anomalies

34. Anomaly Detection • Define function • Represents the within-Class difference of the test activity w.r.t. previous class members • Pick (learn) a particular threshold to detect anomalies

35. Activity Discovery and Anomalous Activity Explanation • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly

36. Anomaly Explanation • Explanatory features: • Consistent • Frequent • Explanation based on features that were: • Deficientfrom an anomaly but were frequently and consistentlypresent in regular members • Extraneous in an anomaly but consistently absentfrom the regular members

37. Anomaly Explanation • Explanatory features: • Consistent • Frequent • Explanation based on features that were: • Deficientfrom an anomaly but were frequently and consistentlypresent in regular members • Extraneous in an anomaly but consistently absentfrom the regular members

38. Anomaly Explanation • Explanatory features: • Consistent • Frequent • Explanation based on features that were: • Deficientfrom an anomaly but were frequently and consistentlypresent in regular members • Extraneous in an anomaly but consistently absentfrom the regular members

39. Anomaly Explanation

40. Experimental Setup – Loading Dock • Barns & Nobel Loading Dock Area • One month worth of data: • 5 days a week – 9 a.m. till 5 p.m. • Event Vocabulary – 61 events • 195 activities: • 150 train activities + 45 test activities Bird’s Eye View of Experimental Setup

41. Results General Characteristics of Discovered Activity Classes • UPS Delivery Vehicles • Fed Ex Delivery Vehicles • Delivery Trucks – multiple packages delivered • Cars and vans, only 1 or 2 packages delivered • Motorized cart used to pick and drop packages • Van deliveries – no use of motorized cart • Delivery trucks – multiple people

42. Results General Characteristics of Discovered Activity Classes Few of the detected Anomalies • UPS Delivery Vehicles • Fed Ex Delivery Vehicles • Delivery Trucks – multiple packages delivered • Cars and vans, only 1 or 2 packages delivered • Motorized cart used to pick and drop packages • Van deliveries – no use of motorized cart • Delivery trucks – multiple people • Back door of delivery not closed • (b) More than usual number of people • involved in unloading • (c) Very few vocabulary events performed

43. Results • Are the detected anomalous activities ‘interesting’ from human view-point? Anecdotal Validation: • Studied 7 users • Showed each user 8 regular activities selected randomly • Showed each user 10 test activities, 5 regular and 5 detected anomalous activities • 8 out of 10 activity-labels of the users matched the labels of our system • Probability of this match happening by chance is 4.4%

44. Experimental Setup – House Environment • House environment – Commercially available strain gages • Five month worth of daily data (151 days): • Event Vocabulary – 16 events • 151 activities Top View of Experimental Setup

45. Visualization of Discovered Clusteres 1 2 5 3 4

46. Activity Discovery and Anomalous Activity Explanation - Recap • Anomaly - “deviation” from the “common” or “regular” • Key Questions: • ‘representation’ of activities • ‘regular’ activities • ‘different’ from regular • ‘explain’an anomaly

47. Hard Question(s) • Importance of semantically meaningful activity-classes? • If not – can we construct a rules to translate computer-discovered classes to something human interpretable?