Body detection, tracking and analysis E-TEAM

1. Body detection, tracking and analysisE-TEAM • Participants (9): • FORTH, ACV, BILKENT, SZTAKI, ICG, University of Amsterdam, University of Surrey, Technion, UPC • E-Team Leader: Montse Pardàs (Cristian Cantón) (UPC)

2. Participants • FORTH: Antonis Argyros, Panos Trahanias • ACV: Herbert Ramoser • Bilkent: Ugur Gudukbay, Enis Cetin, Yigithan Dedeoglu, B. Ugur Toreyinç • SZTAKI: Tamas Sziranyi • ICG: Horst Bischof • University of Amsterdam: Thang Pham, Michiel van Liempt, Arnold Smeulders • University of Surrey: Bill Christmas • Technion/MM: E.Rivlin, M. Rudzsky • UPC: Montse Pardas, Jose Luis Landabaso, Cristian Canton

3. Description • Relevant to WP5 (Single modality processing) and WP11 (Integration and Grand Challenges: Detecting and interpreting humans and human behaviour in videos) • Objective: To increase collaboration in: • Body detection. Using for instance background learning techniques in both single and multi-camera environments. Persons will be identified by means of classification techniques. • Body tracking. By means of models (e.g., templates, 3D models, classifiers) and appropriate motion prediction. • Body analysis. Body models are being used for analysis and tracking. They can range from simple to complex models, depending on the applications.

4. UPC application: smart rooms • Object localization and tracking task in indoor environments surveyed by multiple fixed cameras

5. UPC: Detection and tracking • The method uses a foreground separation process at each camera, based on Stauffer and Grimson background learning • A 3D-foreground scene is modeled and discretized into voxels making use of all the segmented views • Voxels are grouped into blobs • Color information together with other characteristic features of 3D object appearances are temporally tracked using a template-based technique

6. UPC: 3D Blob Extraction

7. UPC: Body and gesture analysis • Aim: obtain the body posture of several people present in a room. • Many pattern analysis challenges can be addressed in this framework: • Gesture analysis: • Scence understanding and classification (who is doing what? i.e. someone raises his hand to ask a question) • Friendly and non-intrusive Human Computer Interfaces (HCI) • Gait analysis: • Biometrics • Motion disorders detection and diagnosis

8. UPC: Model based analysis • Aim: Extract the posture of a human body based on a hierarchical representation of its skeleton.

9. Simple body model Position analysis, simple body action (standing up, walking,…). Example (I) – Simple Model

10. Example (II) – Not so simple model Related publications: • C.Canton-Ferrer, J.R.Casas, M.Pardàs, Towards a Bayesian Approach to Robust Finding Correspondences in Multiple View Geometry Environments, CGGM, Atlanta (USA). LNCS 3515:281-289, Springer-Verlag, 2005. • C.Canton-Ferrer, J.R.Casas, M.Pardàs, Projective Kalman Filter: Multiocular Tracking of 3D Locations Towards Scene Understanding, MLMI, Edinburgh (UK). To appear in LNCS, 2005.

11. Stick body model Gesture analysis, Gait analysis, Biometrics,… Example (III) – Skeleton model • From the voxels data-set, extract information about the structure and the position of the joints of our skeleton model.

12. UPC • Possible collaboration: • Introduce classification of the detected objects in the smart-room context • Introduce new techniques of gesture or activity recognition in the smart room context • Support other groups in the extension from single camera to multi camera • Introduce body models in other groups applications • New applications/analysis methods over our data (availability to generate multi-camera data)

13. University of Amsterdam • Reconstruction of trajectories of people in street surveillance videos: • People detection: state-of-the-art from literature • Tracking algorithm: our own work with solid software implementation • People matching: our own work in general object matching with color invariant descriptors (software is still under development)

14. Sztaki (Szriyanzi) • Aim: Extraction of simple biometric motion of walking and human actions from videos • Method: • The method works with spatio-temporal input information to detect and classify typical patterns of human movement. • Real-time operations • New information-extraction and temporal-tracking method based on a simplified version of the symmetry pattern extraction, which pattern is characteristic for the moving legs of a walking person. This pattern also helps in recognizing human events of more people and unusual actions.

15. Symmetry patterns of walking humans

16. Feature extraction – Identification of the leading leg Leading leg: the “staning” leg from 2 steps, Ratio of integrated leg-areas d

17. Sztaki (Chetverikov) • Robust Structure-from-Motion, 3D motion segmentation and grouping • Given a set of feature points tracked over the frames, we can do robust SfM in presence of more than 50% outliers. • Based on that, we can do robust 3D motion segmentation of multiple objects in presence of occlusion, outliers, and for moving camera. • Recently, we have also developed a novel method for grouping the segmented parts, in order to decide which of them are related. For example, one can determine if an object rotates around an axis defined by another object.

18. Bilkent • Human body extraction, tracking and activity recognition from video sequences. • Body detection and extraction based on motion detection and object shape based classification techniques, background learning and silhouette shape-based object classification. • Multi-person and single person tracking: Correspondence-based whole body tracking and model-based body part tracking methods. • Human action recognition: Tracking results will be combined with activity models (action templates), Hidden Markov models and dynamic programming techniques.

19. ACV • Fast Spatio-Temporal tracking based on Principal Curves

20. ACV • Back-projected reconstructed trajectories

21. ACV • Possible cooperation: • Applying our tracking methods to your data • Benchmarking, evaluation of motion detection, tracking performance • Algorithms for fast computation of informative descriptors (for recognition and tracking tasks)

22. ICG • People detection based on an On-line Adaboost method, which is embedded in a learning framework that can train a Person detector without hand labelling. • Appearance based tracking of people based on an on-line classifier. • Both methods are based on integral orientation histogram features and are able to run in real-time on a standard PC

23. ICG • Possible contributions: • Various sequences we use for testing our methods (some of them with ground truth). • Combining our methods with other techniques to improve the robustness and applicability.

24. Technion • Detection of moving objects • Tracking of detected targets • Classification to one of predefined classes: • human, • human group, • animal, • Vehicle E.Rivlin,M.Rudzsky, R.Goldenberg, U.Bogolmolov and S.Lapchev.,ICPR'02 Y.Bogomolov, G.Dror, S.Lapchev, E.Rivlin, M.Rudzsky. BMVC’03

25. The classes handled by the system

26. walk run run45 Technion • Classification of moving objects: • Single human (walking, running, crawling) Tracking

27. Technion • Possible collaboration in research of human body detection, tracking and motion analysis in multi-camera environments

28. University of Surrey • Automated Audio-Visual Analysis • Work on recognition of activities in the context of sports videos and visual surveillance. We are concentrating on 2-D analysis, using shape and motion cues. • Also work on 3-D representations for human activity recognition. • We have available a public domain (LGPL) C++ library that includes a good framework for integrating different types of video sources & outputs

29. Example

30. FORTH • FORTH has developed a hand detector and tracker which: • Handles multiple, potentially occluding blobs • Supports detection of the fingers of hands • Provides 3D information for the contours of the tracked blobs • Operates with potentially moving cameras • Robust performance under considerable illumination changes • Real time performance (>30fps) • Has already been employed in many applications (cognitive interpretation of human activities, a prototype human-computer interaction system, landmark detection in robot navigation experiments, etc)

31. FORTH - Example

32. FORTH • On-going and future research activities: • Investigation of the use of additional cues (motion, shape, etc) for model-based human motion detection and tracking • Development of inference mechanisms to handle missing parts and uncertain detection estimates • Research in gesture recognition and human activity interpretation

33. E-team possible cooperation • Main outcome of e-teams: joint research papers! • Ideas: • Extend methods developped for single camera to multiple-camera applications • Exchange databases / applications • Extend systems using tools from other groups • Create sub-groups for: • Body detection • Body tracking • Object classification • Gesture analysis

34. E-team possible cooperation • How? • Software exchanges (executables, code, …) • Students visits • Two weeks, financed by MUSCLE • A few months, with student grants • Create a list: who wants to host or send someone in a very specific subject • For every sub-group publish on the Muscle web the on-going collaborations (title, partners, results…)

35. Face detection and recognitionE-TEAM • Participants (3): • ICG, AUTH, UPC • E-Team Leader: Not decided yet (M.Pardàs, C. Cantón) (UPC) • Note: If this E-TEAM is too small it could be embedded into the Body E-TEAM

36. ICG • Researchers: Horst Bischoff • Face detection, tracking and recognition based on local orientation histograms • On-line Adaboost algorithm as an algorithm to cope with all this tasks • Real time operation

37. AUTH • Researchers: Ionnis Pitas, Nikos Nikolaidis • Expertise in face detection, tracking and verification based on several detection techniques developed for greyscale and color images. • Techniques based on morphological elastic graph matching.

38. UPC • Researchers: Ferran Marqués, Verónica Vilaplana

39. Measures & Soft Classifiers Node Extension Hard Classifiers Measures Classifier Final Decision Perceptual Model Perceptual Model Basic Descriptors Basic Descriptor #1 Basic Descriptor #2 … Basic Descriptor #M Shape Descriptor Specific Descriptors Specific Descriptor #1 Specific Descriptor #2 … Specific Descriptor #N BPT Region

40. Frontal Face: Perceptual Model • Candidate selection (in maroon): • Non-complete representation of the object. • Shape descriptor (in orange): • Union of regions that may not be linked in the BPT.