380 likes | 478 Views
DARPA. Rama Chellappa, Yiannis Aloimonos, Doug Ayers, Ross Cutler, Larry Davis, Azriel Rosenfeld, Chandra Shekhar University of Maryland. Activity Monitoring October 19-20, 1999. Bob Bolles, Brian Burns, Marty Fischler, Ravi Gopalan, Marsha Jo Hannah, Dave Scott SRI International.
E N D
DARPA Rama Chellappa, Yiannis Aloimonos, Doug Ayers, Ross Cutler, Larry Davis, Azriel Rosenfeld, Chandra Shekhar University of Maryland Activity Monitoring October 19-20, 1999 Bob Bolles, Brian Burns, Marty Fischler, Ravi Gopalan, Marsha Jo Hannah, Dave Scott SRI International
Application Challenge Develop techniques for dramatically increasing the productivity of an aerial video analyst.
High-Level Approach Sensor Multiplexing to Monitor Several Sites “Simultaneously” and Semi-automatically
Technical Goalfor Activity Monitoring Develop techniques to monitor sites, such as cantonment areas and tree lines, from an airborne platform and identify tactically significant activities involving people and vehicles. • Sample Activities: • people entering a forbidden area • people congregating near an embassy • vehicles convoying along a road • people readying a missile for launch
Activity Template Starting search Zoom to a NFOV & aim close to tree line Looking for people Move to new point along tree line Detect person(s) Looking for large vehicle Detect small vehicle All people leave the FOV Detect large vehicle Exit of large vehicle detected Technical Challengesfor Activity Monitoring • Representation of activities • Recognition of activities from a moving platform • Moving object classification Activity A large tactical vehicle exiting a hide site (along a tree line). People are often visible guiding the vehicle out.
Approach Task specification • Retrieve or sketch a site model (roads, buildings,…) • Specify the task (what, where, when, & reports/alarms) Automatic monitoring • Scan the appropriate area • Stabilize the video (MTS -- Sarnoff) • Register the video to the site model (PVR -- Harris) • Detect and track moving objects • Characterize & classify the tracked objects • Recognize activities • Report tactically significant events AMIS -- Activity Monitoring Integrated Systesm
“Residence” Area Berm Motorpool Powers Road Mosby Road Site Model Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events
Drivers jog to their vehicles Drivers jog to their vehicles Vehicles drive away Task Specification Residence Area Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Motorpool
Sensor Field of View Scan Area Residence Area Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Motorpool
Stabilize Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Raw Video
Stabilize Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Stabilized Video
Actual field of view Register Video Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Desired field of view
Track Objects Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events
Characterize Objects Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events • Object Properties • Size, velocity, … • Articulation -- periodicity • (for animate/inanimate) • Could it be parallax? • Color, shape, … • Location in the site
People moving down Powers Road People approaching motorpool area People entered motorpool area Vehicles leaving motorpool area Report Events Site model Task specification Scan area Stabilize video Register video Track objects Characterize objects Recognize activities Report events Alert: Battle Group Pullout
Primary Contributions • Representation and recognition of activities (in the context of a site model) • augmented finite state machines • dynamic belief networks • Moving object classification components • parallax analysis • animate/inanimate classification • velocity, size, ...
Activity Monitoring “Residence” Area 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities Berm Motorpool
Activity Template Starting search Zoom to a NFOV & aim close to tree line Looking for people Move to new point along tree line Detect person(s) Looking for large vehicle Detect small vehicle All people leave the FOV Detect large vehicle Exit of large vehicle detected Activity Templates Event Primitives • Approaching/Leaving • Gaining-Ground/ Losing-Ground • Entering/Exiting • Moving-inside-region • Temporal durations Combinations • Boolean operations • Sequences • Graphs
Video Registration Image World
Activity Analysisin World Coordinates Image World
Moving Object Detection Raw video fields Image N Raw differences AND’d differences
Animate/Inanimate Periodicity analysis
Periodicity Analysisfor Classifying Objects as Animate or Inanimate Track objects Align and scale objects Compute similarity matrix S Autocorrelate S Template fit peaks of S
Parallax Detection Flagged as being locally consistent with “motion parallax”
Air-Ground Partition for 1999 Multiple Target Surveillance CAGS-Air Raw Video (analog) Metadata Stabilization Params Ground Station Activity Monitoring CAGS-Ground Precision Video Registration MTS-Ground
Drivers jog to their vehicles Drivers jog to their vehicles Vehicles drive away Battle Group Pullout 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road Activities
Drivers walk back to residence Vehicles return & park Battle Group Return Activities 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road
People Exiting Woods near Berm Activities 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road People Exit Tree Line
People Exit Tree Line and cross the road People Crossing Road Activities 1. Battle group pullout 2. Battle group return 3. People exiting woods near berm 4. People crossing the road
PreliminaryEvent Statistics • Results from 2 flights with high contrast imagery
PreliminaryWhole Vignette Statistics • Results from 2 flights with high contrast imagery
Summary Increase the productivity of an image analyst by a factor of 10 to 15 by multiplexing a high-performance sensor and automatically identifying potentially significant activities. Goal: • Accomplishments: • AMIS – Activity Monitoring Integrated System • Activity Templates – an initial representation for activities • An initial technique for recognizing activities based on augmented finite state machines • An extension to dynamic belief networks to activity recognition • A technique for identifying moving objects due to motion parallax • A technique for classifying moving objects as animate or inanimate • A semi-automatic video registration technique • A realtime moving object detection technique
Evaluation of‘99 Accomplishments • Moving object classification -- Components only • Sensor Control -- manual versus computer-controlled • HCI -- primarily on PC, not integrated into CAGS-Ground
Plans for ‘00 • Represent & recognize more complex activities, such as checkpoint monitoring • Call PVR for video registration • Place sensor under computer-control (based on MTS results) • Integrate moving object classification • Integrate the HCI into CAGS-Ground