Activity Monitoring October 19-20, 1999

1. DARPA Rama Chellappa, Yiannis Aloimonos, Doug Ayers, Ross Cutler, Larry Davis, Azriel Rosenfeld, Chandra Shekhar University of Maryland Activity Monitoring October 19-20, 1999 Bob Bolles, Brian Burns, Marty Fischler, Ravi Gopalan, Marsha Jo Hannah, Dave Scott SRI International

2. Application Challenge Develop techniques for dramatically increasing the productivity of an aerial video analyst.

3. High-Level Approach Sensor Multiplexing to Monitor Several Sites “Simultaneously” and Semi-automatically

4. Technical Goalfor Activity Monitoring Develop techniques to monitor sites, such as cantonment areas and tree lines, from an airborne platform and identify tactically significant activities involving people and vehicles. • Sample Activities: • people entering a forbidden area • people congregating near an embassy • vehicles convoying along a road • people readying a missile for launch

5. Activity Template Starting search Zoom to a NFOV & aim close to tree line Looking for people Move to new point along tree line Detect person(s) Looking for large vehicle Detect small vehicle All people leave the FOV Detect large vehicle Exit of large vehicle detected Technical Challengesfor Activity Monitoring • Representation of activities • Recognition of activities from a moving platform • Moving object classification Activity A large tactical vehicle exiting a hide site (along a tree line). People are often visible guiding the vehicle out.

6. Approach Task specification • Retrieve or sketch a site model (roads, buildings,…) • Specify the task (what, where, when, & reports/alarms) Automatic monitoring • Scan the appropriate area • Stabilize the video (MTS -- Sarnoff) • Register the video to the site model (PVR -- Harris) • Detect and track moving objects • Characterize & classify the tracked objects • Recognize activities • Report tactically significant events AMIS -- Activity Monitoring Integrated Systesm

7. “Residence” Area Berm Motorpool Powers Road Mosby Road Site Model Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events

8. Drivers jog to their vehicles Drivers jog to their vehicles Vehicles drive away Task Specification Residence Area Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Motorpool

9. Sensor Field of View Scan Area Residence Area Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Motorpool

10. Stabilize Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Raw Video

11. Stabilize Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Stabilized Video

12. Actual field of view Register Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Desired field of view

13. Track Objects Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events

14. Characterize Objects Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events • Object Properties • Size, velocity, … • Articulation -- periodicity • (for animate/inanimate) • Could it be parallax? • Color, shape, … • Location in the site

15. People moving down Powers Road People approaching motorpool area People entered motorpool area Vehicles leaving motorpool area Report Events Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Alert: Battle Group Pullout

16. Primary Contributions • Representation and recognition of activities (in the context of a site model) • augmented finite state machines • dynamic belief networks • Moving object classification components • parallax analysis • animate/inanimate classification • velocity, size, ...

17. Introduction toLive Flight Experiments

18. Activity Monitoring “Residence” Area 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities Berm Motorpool

19. Activity Template Starting search Zoom to a NFOV & aim close to tree line Looking for people Move to new point along tree line Detect person(s) Looking for large vehicle Detect small vehicle All people leave the FOV Detect large vehicle Exit of large vehicle detected Activity Templates Event Primitives • Approaching/Leaving • Gaining-Ground/ Losing-Ground • Entering/Exiting • Moving-inside-region • Temporal durations Combinations • Boolean operations • Sequences • Graphs

20. Site Model Sketching

21. Video Registration Image World

22. Activity Analysisin World Coordinates Image World

23. Moving Object Detection Raw video fields Image N Raw differences AND’d differences

24. Parallax Versus Independent Motion

25. Animate/Inanimate Periodicity analysis

26. Periodicity Analysisfor Classifying Objects as Animate or Inanimate Track objects Align and scale objects Compute similarity matrix S Autocorrelate S Template fit peaks of S

27. Parallax Detection Flagged as being locally consistent with “motion parallax”

28. AM’s Windows

29. Air-Ground Partition for 1999 Multiple Target Surveillance CAGS-Air Raw Video (analog) Metadata Stabilization Params Ground Station Activity Monitoring CAGS-Ground Precision Video Registration MTS-Ground

30. Drivers jog to their vehicles Drivers jog to their vehicles Vehicles drive away Battle Group Pullout 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities

31. Drivers walk back to residence Vehicles return & park Battle Group Return Activities 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road

32. People Exiting Woods near Berm Activities 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road People Exit Tree Line

33. People Exit Tree Line and cross the road People Crossing Road Activities 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road

34. PreliminaryEvent Statistics • Results from 2 flights with high contrast imagery

35. PreliminaryWhole Vignette Statistics • Results from 2 flights with high contrast imagery

36. Summary Increase the productivity of an image analyst by a factor of 10 to 15 by multiplexing a high-performance sensor and automatically identifying potentially significant activities. Goal: • Accomplishments: • AMIS – Activity Monitoring Integrated System • Activity Templates – an initial representation for activities • An initial technique for recognizing activities based on augmented finite state machines • An extension to dynamic belief networks to activity recognition • A technique for identifying moving objects due to motion parallax • A technique for classifying moving objects as animate or inanimate • A semi-automatic video registration technique • A realtime moving object detection technique

37. Evaluation of‘99 Accomplishments • Moving object classification -- Components only • Sensor Control -- manual versus computer-controlled • HCI -- primarily on PC, not integrated into CAGS-Ground

38. Plans for ‘00 • Represent & recognize more complex activities, such as checkpoint monitoring • Call PVR for video registration • Place sensor under computer-control (based on MTS results) • Integrate moving object classification • Integrate the HCI into CAGS-Ground