vision-based activity recognition systems

1. vision-based activity recognition systems Ali Taalimi Prof. Abidi

2. Outline • General Overview/Application • Security and Surveillance • Surveillance Applications • General framework of visual surveillance • Long Range Face Recognition

3. application of vision-based activity recognition systems • Surveillance • Automatic recognition of anomalies, searching for an activity of interest in a large database by learning patterns of activity from long videos • Ubiquitous cameras in public places (e.g. CCTVs). • In London, an average person is monitored 300 times / day. • Goal • Monitor suspicious activities for real-time reactions. • ‘Fighting’, ‘stealing’. • Currently, surveillance systems are mainly for recording. • Activity recognition is essential for surveillance and other monitoring systems in public places

4. application of vision-based activity recognition systems • Intelligent environments (HCI) • Understanding the interaction between a computer and a human specially for human computer interfaces (smart rooms). • Intelligent home, office, and workspace • Monitoring of elderly people and children. • Support one’s quality of life. • Recognition of ongoing activities and understanding of current context is essential.

5. application of vision-based activity recognition systems Content Based Video Analysis: learning of patterns from raw video and summarizing a video based on its content Web-based video retrieval: Example: search ‘kick’ from long movies Sports play analysis

6. Levels of human activities Gestures: Atomic movements/Single body-part movements Actions: A single actor/Single actor movements Interactions: Human-human interactions Human-object interactions Group activities: Physical/conceptual groups

7. Challenges • robustness • Environment variations: • Background • Moving backgrounds – trees • Pedestrians: Occlusions • View-points: moving camera • Actor movement variations: Each person has his/her own style of executing an activity • various activities • There are various types of activities: • The ultimate goal is to make computers recognize all of them reliably.

8. General Overview A generic action or activity recognition system is proceeding from a sequence of images to a higher level interpretation in a series of steps

9. Outline General Overview/Application Security and Surveillance Surveillance Applications General framework of visual surveillance Long Range Face Recognition

10. Security and Surveillance • processing framework of visual surveillance in dynamic scenes includes: • modeling of environments • detection of motion • classification of moving objects • tracking • understanding and description of behaviors • human identification • fusion of data from multiple cameras (overcome the occlusion) • Visual surveillance in dynamic scenes attempts to detect, recognize and track certain objects from image sequences, and to understand and describeobject behaviors. • Mounting video cameras is cheap, but human resources to observe the output is expensive.

11. Security and Surveillance • Possible research directions: • occlusion handling, • a combination of two and 3D tracking, • a combination of motion analysis and biometrics, • anomaly detection and behavior prediction, • content-based retrieval of surveillance videos, • behavior understanding • fusion of information from multiple sensors, • and remote surveillance.

12. Surveillance Applications • Access control in special areas • uses the visitor’s features (height, facial appearance and walking gait) • 2) Person-specific identification in certain scenes • police build a biometric feature database of suspects, and place visual surveillance systems at locations where the suspects usually appear (subway stations, casinos). The systems automatically recognize and judge whether people in view are suspects from distance. • 3) Crowd flux statistics and congestion analysis • traffic management: monitor expressways and junctions of the road network, and further analyze the traffic flow and the status of road congestion • 4) Anomaly detection and alarming • analyze the behaviors of people and vehicles and determine whether these behaviors are normal or abnormal (parking lots and supermarkets) • 5) Interactive surveillance using multiple cameras • for social security, multiple cameras could be used to ensure the security over a wide area

13. Visual surveillance large research projects • DARPA in 1997: • Visual Surveillance and Monitoring (VSAM) • develop automatic video understanding technologies that enable a operator to monitor behaviors over complex areas such as battlefields and civilian scenes. • DARPA in 2000: • the Human Identification at a Distance (HID) • enhance protection from terrorist attacks, • Use multimodal surveillance technologies for detecting, classifying, and identifying humans at great distances.

14. Outline • General Overview/Application • Security and Surveillance • Surveillance Applications • General framework of visual surveillance • Long Range Face Recognition

15. General framework of visual surveillance • companies like Sony and Intel have designed equipment suitable for visual surveillance, 1) active cameras, 2) smart cameras, 3) omni-directional cameras …

16. General framework of visual surveillance Motion Detection and Object Tracking every visual surveillance system starts with motion detection. Definition: segmenting regions corresponding to moving objects from the rest of an image. It includes: Environment Modeling, Motion Segmentation, Object Classification Environment Modeling: recover and update background images from a dynamic sequence (2D/3D) Motion segmentation: detect regions corresponding to moving objects like vehicles and humans. Object Classification: Different moving regions = different moving targets in natural scenes. Exm: road traffic scenes probably include humans

17. UNDERSTANDING AND DESCRIPTION OF BEHAVIORS Behavior Understanding After tracking the moving objects from one frame to another. behaviors is: classification of time varying feature data, i.e., matching an unknown test sequence with a reference sequences of typical behaviors. fundamental problem: 1) Learning the reference behavior sequences from training samples. 2) Elaborate both training and matching methods to cope with small variations. Human activity: A collection of human/object movements with a particular semantic meaning Activity recognition: Finding of video segments containing such movements

18. Personal Identification For Visual Surveillance • The problem of “who is now entering the area under surveillance” • Most interested features: Human face and gait detection. • The main steps in the face recognition for visual surveillance are face detection, face tracking, face feature detection and face recognition • major methods for gait recognition: • Statistical Methods, Physical-Parameter-Based Methods, Spatio-Temporal Motion-Based Methods • Fusion of Gait With Other Biometrics

19. FUSION OF DATA FROM MULTIPLE CAMERAS • Why multiple camera-based visual surveillance systems: • the surveillance area is expanded • multiple view information can overcome occlusion • Problems: • camera installation, camera calibration, object matching, automated camera switching, and data fusion. • camera installation: • how to cover the entire scene with the minimum number of cameras. • Redundant cameras increase processing time, complexity and installation cost • lack of cameras cause blind angles, reduce reliability of surveillance system. • object matching: • Finding the correspondences between the objects in different image sequences taken by different cameras. • automated camera switching: • system switch to the camera that may give a better view of the object. problems: find the better camera + minimizing the number of switches.

20. Outline General Overview/Application Security and Surveillance Surveillance Applications General framework of visual surveillance Long Range Face Recognition

21. Motivating Cooperative Face Controlled pose/position and lighting Non-Cooperative Face No control over subject Nighttime?

22. Persons Identification at a Distance with 3D Face Modeling • Most biometric features (such as fingerprints, hand shape, iris or retinal scans) require cooperative subjects • The goal is to identify non-cooperative individuals at a distance from a sequence of images using 3D face model. • advantages of using 3D for recognition are pose and lighting variation compensation. • 3D face recognition using active 3D range sensors is appropriate for close distance (use image sequence instead) • three steps approach: • keyframe detection • camera motion estimation • multiple view dense stereo matching

23. Long Range Face Recognition A Passive Stereo System for 3D Human Face Reconstruction and Recognition at a Distance, CVPRW, 2012 To address pose and illumination, researchers recently are focusing on 3D face recognition 3D face geometry obtain by 3D sensing devices such as laser scanners or reconstructed from one or more images. constructed their own passive stereo acquisition setup.

24. cont reconstruction