Motivation

1. Motivation Where is my W-2 Form?

2. Video-based Tracking Camera Camera view of the desk Overhead video camera

3. Example 40 minutes, 1024x768 @ 15 fps

4. System Overview User Input Video

5. System Overview Internal Representation User Input Video Desk Desk Video Analysis T T+1 Vision Engine

6. System Overview Internal Representation Query (where is my W-2 form?) User Input Video Desk Desk Video Analysis T T+1 Query Interface

7. System Overview Internal Representation Query (where is my W-2 form?) User Input Video Desk Desk Video Analysis Answer T T+1 Query Interface

8. System Overview Internal Representation User Input Video Desk Desk Video Analysis T T+1 Vision Engine

9. … … Vision Problem

10. … … Vision Problem Event

11. … … Vision Problem Event … Desk

12. … … Vision Problem Event … … Desk Desk

13. … … Vision Problem Event sanders01.pdf lowe04sift.pdf tut-article.pdf … … objectspaces.pdf kidd94.pdf Desk Desk

14. … … Vision Problem Event Scene Graph (DAG) sanders01.pdf lowe04sift.pdf tut-article.pdf … … objectspaces.pdf kidd94.pdf Desk Desk

15. Assumptions • Simplifying • Corresponding electronic copy exists

16. Assumptions • Simplifying • Corresponding electronic copy exists • 3 event types: move/entry/exit • One document at a time • Only topmost document can move • No duplicate copies of same document

17. Assumptions • Simplifying • Corresponding electronic copy exists • 3 event types: move/entry/exit • One document at a time • Only topmost document can move • No duplicate copies of same document • Other • Desk need not be initially empty

18. Algorithm Overview Input Frames … … Event Detection Event Interpretation Document Recognition Scene Graph Update

19. Algorithm Overview Input Frames … … Event Detection before after Event Interpretation Document Recognition Scene Graph Update

20. Algorithm Overview Input Frames … … Event Detection before after Event Interpretation “A document moved from (x1,y1) to (x2,y2)” Document Recognition Scene Graph Update

21. Algorithm Overview Input Frames … … Event Detection before after Event Interpretation “A document moved from (x1,y1) to (x2,y2)” File1.pdf Document Recognition File2.pdf File3.pdf Scene Graph Update

22. Algorithm Overview Input Frames … … Event Detection before after Event Interpretation “A document moved from (x1,y1) to (x2,y2)” File1.pdf Document Recognition File2.pdf File3.pdf Scene Graph Update

23. Algorithm Overview Input Frames … … Event Detection before after Event Interpretation “A document moved from (x1,y1) to (x2,y2)” File1.pdf Document Recognition File2.pdf File3.pdf Scene Graph Update

24. … … Event Detection

25. … … Event Detection Frame Difference time

26. … … Event Detection Event Frames Frame Difference Threshold time time

27. … … Event Detection Event Frames before after

28. Algorithm Overview Input Frames … … Event Detection before after Event Interpretation “A document moved from (x1,y1) to (x2,y2)” File1.pdf Document Recognition File2.pdf File3.pdf Scene Graph Update

29. Event Interpretation before after Move

30. Event Interpretation before after Move Entry

31. Event Interpretation before after Move Entry Exit

32. Event Interpretation before after Move Motion: (x,y,θ) Entry Exit

33. Event Interpretation before after Move 1. Move vs. Entry/Exit Entry Exit

34. Event Interpretation before after Move Entry 2. Entry vs. Exit Exit

35. Event Interpretation • Use SIFT [Lowe 99] • Scale Invariant Feature Transform • Distinctive feature descriptor • Reliable object recognition

36. Move vs. Entry/Exit after before

37. Move vs. Entry/Exit after before

38. Move vs. Entry/Exit after before

39. Move vs. Entry/Exit after before

40. Move vs. Entry/Exit after before

41. Move vs. Entry/Exit after before

42. Move vs. Entry/Exit after before

43. Move vs. Entry/Exit after before

44. Move vs. Entry/Exit after before

45. Move vs. Entry/Exit after before

46. Move vs. Entry/Exit after before

47. Move vs. Entry/Exit Motion: (x,y,θ) after before

48. Entry vs. Exit Example 1 (entry) before after

49. Entry vs. Exit Example 1 (entry) before after

50. Entry vs. Exit Example 1 (entry) before after