Attribute Multiset Grammars for Global Explanations of Activities

1. Attribute Multiset Grammars for Global Explanations of Activities Dima Damen, David Hogg Computer Vision Group

2. Activity Recognition

3. Activity Recognition

4. Activity Recognition

5. Activity Recognition

6. Activity Recognition

7. Activity Recognition

8. Contribution • Define global explanations using Attribute Multiset Grammars • Parsing a set of detections recognition • Bayesian approach to finding the best parse tree • Two case studies

9. Activity Recognition Event Definition 1 event

10. Activity Recognition Event Definition Event Threads ?

11. Activity Recognition Event Definition Global Explanation

12. Activity Recognition • Uncertain detections • Multiple definitions

13. Activity Recognition < = time pos time pos • Intra-activity constraints • Temporal Constraints • Spatial Constraints • Other Geometric Constraints • Inter-activity constraints

14. Activity Recognition • Complex events

15. Definition using AMG

16. Definition using AMG T = { , , } D A E b a c N = { , , } Synthetic Rule Attribute Rule Attribute Constraints A → a, b … a.time < b.time

17. Definition using AMG Synthetic Rule Attribute Rule Attribute Constraints A → a, b … a.time < b.time E → a, c c.count < 1 D → A, E ….

18. Definition using AMG 2 3 1 1 1 2 2 4 3 2 3 1 1 1 2 2 3 4

19. Definition using AMG 1 1 Synthetic Rule Attribute Rule Attribute Constraints A → a, b A.dist = a.pos – b.pos a.time < b.time 2 3 1 1 1 2 2 4 3 2 3 1 1 1 2 2 3 4

20. Definition using AMG T = { , , } S A D E b c a D D D N = { , , } A E A A E c2 a1 b1 c1 a2 a4 b3 a2 b2 a3 1 1 1 2 4 3 2 2 2 3 2 3 1 1 1 2 2 4 3

21. Definition using AMG detections • Attribute Multiset Grammars G = (N, T, S, A, P) Multiset

22. Recognition using AMG D = {a1, a2, b1, c1, c2}

23. Recognition using AMG D = {a1, a2, b1, c1, c2} Algorithm in Paper

24. Recognition using AMG D = {a1, a2, b1, c1, c2}

25. Recognition using AMG D = {a1, a2, b1, c1, c2}

26. Recognition using AMG Searching the space of explanations [CVPR 09] • Greedy Search • Multiple Hypotheses Tree [BMVC 07] • Reversible Jump Markov Chain Monte Carlo

27. Case I : The Bicycles Problem

28. The Bicycles Problem Dropping Picking Pass-by Drop Pick Pick-Drop

29. The Bicycles AMG

30. Case II: The Enter-Exit Problem

31. Global Explanation time • Global explanation

32. Global Explanation time

33. Using Baggage Detector [ECCV 08]

34. Selected Features 1. Matching Heights

35. Selected Features 2. Clothing Colour

36. Selected Features 3. Baggage Colour

37. Selected Features 4. Baggage Relative Height

38. Experiment • 12 hours • 326 people • 429 candidate bags • 62 ground truth pairs

39. Results

40. Conclusion • Defining activity using AMG • Hierarchies of events • Multisets • Intra-activity constraints → synthetic attributes • Inter-activity constraints → inherited attributes • Finding the best parse tree → Recognition • Building BN • Searching for MAP • Two case studies • Future work

41. Thank you 