Multi-Group Tracking with Adaptive Target Model

1. Multi-Group Tracking with Adaptive Target Model PhD thesis proposal Loris Bazzani, PhD student (XXIV cycle) University of Verona Department of Computer Science

2. Objectives • Compare existing Multi-Target Tracking methods, studying the sampling technique • Propose a new tracking method: Multi-Group Tracking • Model robustly and adaptively the target • Integrate target model with Multi-Group tracking

3. Outline • Introduction • Multi-Target Tracking • Multi-Group Tracking • Target Modeling • Conclusions • State of the art • Open issues • Proposed ideas

4. Introduction (1) • Tracking: spatial and temporal localization of a mobile object in an environment monitored by sensor(s) • Multi-target (MTT): keeping the identity of different targets • Reliable: insensible to noise and occlusions • Application to Automated Surveillance

5. Introduction (2) • Multi-Group Tracking (MGT): • Spatial and temporal localization of groups of objects • Motivations: • Humans prefer to stay in group rather than alone • High-level representation of the relations among the targets • MGT is simpler than MTT in a crowded scenario • MGT can help MTT when occlusions occur

6. Introduction (3) • Multi-Group Tracking (MGT): why is MGT a hard task?

7. Introduction (4) • Target Model: • A general and representative example that summarizes any possible changing of the target • intrinsic variations: pose variation and shape deformation • extrinsic variations: illumination changes, camera movement, and occlusions • Not considering the above variation causes the failure of the tracking [Ross08] • Fit with the re-identification problem

8. Multi-Target Tracking (1) Abstract Formulation [Arulampalam02] • State Space Approach for modeling discrete-time dynamic systems • State: abstract nature of the target • Measurement: “visible” dimensions of the state space Filtering • We observe the real world events as a state by the measurement process • Objective: estimating the state of the system at each instant given the measurements

9. Multi-Target Tracking (2) Data Association [Bar-Shalom87] • The observer has at his disposal a huge amount of measurements • Finding the correct correspondences between measurements and states of the system

10. Multi-Target Tracking (3)

11. Multi-Target Tracking (4) Particle Filter (PF) [Isard01]

12. Multi-Target Tracking (5) State Space Conformation [Isard01] (+) Efficient sampling (-) No interaction modeling [MacCormick00] (+) Implicit interaction modeling (-) Curse of dimensionality [Lanz06] (+) Efficient sampling (+) Implicit interaction modeling

13. Multi-Target Tracking (6) - HJS vs. MHT [Bazzani09] - MHT cons: 1) Multiple tracks cause proliferation in the number of tracks 2) Occlusions generate new tracks 3) Not robust to non-linear people motion HJS pros: 1) One track is kept for each target 2) Partial occlusions are handled; 3) Deal with non-linearity of people motion.

14. Multi-Target Tracking (7) • Open issues of PF-based MTT (and MGT): • Sampling Method • Dynamic Model • Linear-Gaussian model • Observation Model • State Estimation • Maximum-A-Posteriori or Weighted Mean

15. Multi-Target Tracking (8.1) Sampling • Sequential importance sampling/re-sampling [Arulampalam02]: classical PF + degeneracy problem avoiding by re-sampling • Regularized PF [Arulampalam02]: resamples applying a Kernel to the particles • MCMC [Andrieu03]: defines a Markov chain over the state space, such that the stationary distribution of the chain is equal to the sought posterior • Reversible-Jump MCMC[Khan05]: switches between variable dimensional state spaces • Rao-Blackwellizing PF [Schindler05]: analytically computes a portion of the distribution other the state space

16. Multi-Target Tracking (8.2) Observation model • Likelihood: compare an observation z given a hypothesis of state of the system x • Usually defined in the Gibbs form: where d is a metric • x and z MUST be represented in the same feature space • ISSUES: • feature space (for x and z) and metric definitions • occlusion handling

17. Multi-Target Tracking (9) - Proposed Research - • Occlusion Handling • Study and implement RJ-MCMC particle filter • Propose a set of jumps in order to cope with tracking a variable number of objects • Propose an observation model in RJ-MCMC framework

18. Multi-Group Tracking (1) (-) Social interactions cannot be caught by the foreground analysis (-) The inter-group dynamic yields a loss of appearance informations defines a group as the moving regions, extracted from a foreground analysis infers the MGT from the tracks estimation carried out by the MOT (e.g. tracks clustering) (-) Direct dependence from MOT (-) The MOT estimation is not reliable when occlusions occur uses the foreground information and MOT to detect groups, but then tracks them as different entities (+) Cancel out the above problems (-) Model creation is a hard task

19. Multi-Group Tracking (2) Foreground-based MGT • Tracking at three levels of abstraction [McKenna00] : • Regions: connected component that have been tracked for T frames • People: one or more regions grouped together • Groups: one or more people grouped together, if they share a region (+) Simplicity (-) heuristic FG analysis

20. Multi-Group Tracking (3) MGT from MOT • MCMC PF for group tracking [Pang07]: • Track groups as ensemble of targets analyzing : group variable • Treated as Bayesian estimation problem • Group structure model: • captures the relations among objects

21. Multi-Group Tracking (4) - Proposed Research - • Problems: • Definition of “Group”: an entity containing targets with similar characteristics (e.g. motion, interactions, ...) • Deterministic/formal definition as an ensemble of objects • Add non-deterministic component into the tracking method • Intra-group occlusions: if we know that the objects hasn’t left the group, we infer that it is still into the group • Inter-group occlusions: tracking of groups

22. Multi-Group Tracking (5) - Proposed Research -

23. Multi-Group Tracking (6) - Proposed Research - • Use a MOT method • Create a MGT method (track only groups) • Definition of group dynamics -> sociological studies • Definition of a group observation model • Define a collaborative probabilistic framework in order to share MGT and MOT informations

24. Target Modeling (1) Train the model using the appearance data available before tracking begins Adapt the model to account for its changes in appearance, using an on-line learning method • Open Issues: • Representation of the target: feature space • Leaning technique

25. Target Modeling (2) Feature • Color Histogram: which is the best color space for tracking? [Sebastian08] • (-) No spatial information • Color correlogram [Huang99], spatiogram [Birchfield05], multi-resolution histogram [Hadjidemetriou01] • (+) Add the spatial information • (-) Increase the computational burden

26. Target Modeling (3) Feature • Covariance descriptors [Porikli06] Spatial and appearance attributes (+) Natural way of fusing multiple features (-) Computationally expensive -> integral images

27. Target Modeling (4) Fixed target models Ensemble of localized features [Gray08] • Define a feature space and let machine learning approach find the best representation • AdaBoost extracts • the object representation: the most discriminative set of features, and • the similarity measures: the most discriminative set of likelihood ratio test • Used for re-identification problem

28. Target Modeling (5) Adaptive target models • Incremental learning of Covariance-based descriptors [Porikli06] • Principal Component Analysis (PCA) incremental learning [Ross08] • Convex combination of models using a learning rate • Feature-based model • Parametric model

29. Target Modeling (6) Patch-based (local) updating [Kwon09] • Evolve photometric and geometric appearance • Local Patches can be added, deleted or moved to different position • Examining the patches by landscape analysis • Bad patches are modified on-line: background patches and patches in regions with high density of patch are deleted • Good patches are moved • Appearance model is updated with a convex combination

30. Target Modeling (7) - Proposed Research - • Part-based multiple-features: • Maximally Stable Color Regions • HS(V) histograms • Recurrent high-structured patches Invariances: view point, partial occlusions illumination, view point, deformations, partial occlusions partial occlusions, illumination Global Local • Temporal updating: • Delete the non-stable features • Cover the Variability of the stable features

31. Conclusions • Compare existing Multi-Target Tracking methods, studying the sampling technique • Propose a new tracking method: Multi-Group Tracking • Model robustly and adaptively the target • Integrate target model with Multi-Group tracking, using HJS and RJ-MCMC

32. The beginning • Now, we “just” put the proposed ideas into practice Thanks for attention Questions?

33. References [Ross08] D.A. Ross, J. Lim, R.S. Lin, and M.H. Yang. Incremental learning for robust visual tracking. International Journal of Computer Vision, 77(1):125–141, 2008. [Arulampalam02] M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans. on Signal Processing, 50(2):174–188, 2002. [Bar-Shalom87] Y. Bar-Shalom. Tracking and data association. Academic Press Professional, Inc., San Diego, CA, USA, 1987. [Isard01] M. Isard and J. MacCormick. Bramble: A bayesian multipleblob tracker. In IEEE Int. Conf. on Computer Vision, 2001. [MacCormick00] John MacCormick and Andrew Blake. A probabilistic exclusion principle for tracking multiple objects. Int. J. Comput. Vision, 39(1):57–71, 2000. [Lanz06] O. Lanz. Approximate bayesian multibody tracking. IEEE Trans. on Pattern Analysis and Machine Intelligence, 28(9):1436–1449, 2006. [Andrieu03] Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. An introduction to mcmc for machine learning. Machine Learning, 50(1):5–43, January 2003.

34. References [Khan05] Z. Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering for tracking a variable number of interacting targets. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(11):1805–1819, 2005. [Schindler05] G. Schindler and F. Dellaert. A Rao-Blackwellized Parts-Constellation Tracker. In ICCV Workshop on Dynamical Vision; International Conference on Computer Vision, 2005. Springer [Mckenna00] Stephen J. Mckenna, Sumer Jabri, Zoran Duric, Harry Wechsler, and Azriel Rosenfeld. Tracking groups of people. Computer Vision and Image Understanding, 2000. [Pang07] Sze Kim Pang, Jack Li, and Simon Godsill. Models and algorithms for detection and tracking of coordinated groups. In Symposium of image and Signal Processing and Analisys, 2007. [Sebastian08] Sebastian, P.; Yap Vooi Voon; Comley, R., "The effect of colour space on tracking robustness," Industrial Electronics and Applications, 2008. ICIEA 2008. 3rd IEEE Conference on , vol., no., pp.2512-2516, 3-5 June 2008 [Huang99] J. Huang, S. Ravi Kumar, M. Mitra, W.J. Zhu, and R. Zabih. Spatial color indexing and applications. International Journal of Computer Vision, 35(3):245–268, 1999. [Birchfield05] ST Birchfield and S. Rangarajan. Spatiograms versus histograms for region-based tracking. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, volume 2, 2005.

35. References [Hadjidemetriou01] E. Hadjidemetriou, MD Grossberg, and SK Nayar. Spatial information in multiresolution histograms. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, volume 1, 2001. [Porikli06] F. Porikli, O. Tuzel, and P. Meer. Covariance tracking using model update based on lie algebra. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, 2006. [Gray08] D. Gray and H. Tao. Viewpoint Invariant Pedestrian Recognition with an Ensemble of Localized Features. In Proceedings of the 10th European Conference on Computer Vision: Part I, pages 262–275. Springer, 2008. [Kwon09] J.S. Kwon and K.M. Lee. Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive basin hopping monte carlo sampling. pages 1208–1215, 2009.