Aleksandar Ivanovic, Thomas S. Huang Backmen Institute for Advanced Science and Technology, UIUC

1. A Probabilistic Framework For Segmentation and Tracking of Multiple Non Rigid Objects for Video Surveillance Aleksandar Ivanovic, Thomas S. Huang Backmen Institute for Advanced Science and Technology, UIUC

2. Outline • Introduction • Pixel probability model • Background model • Foreground probability model • Connected components matching • Object detection • Experimental results • Conclusion

3. Introduction • In video surveillance, reliable segmentation of moving objects is essential for successful event recognition • Park and Aggarwal • A Gaussian mixture model is used at the pixel level to train and classify individual pixel colors • Markov Random Field (MRF) framework is used at the blob level to merge the pixels into coherent blobs and to register inter-blob relations • A coarse model of the human body (head, upper body, lower body) is applied at the object level as empirical domain knowledge to resolve ambiguity due to occlusion and to recover from intermittent tracking failures

4. Introduction • Elgammal and Davis • Use maximum likelihood estimation to estimate the best arrangement for people • Modeling these regions involves modeling their appearance (color distributions) as well as their spatial distribution with respect to the body • Assumption: targets are visually isolated before occlusion so that we can initialize their models • Gomila and Meyer • Each image of a sequence is segmented and represented as a region adjacency graph • Object tracking becomes a particular graph-matching problem, in which the nodes representing the same object are to be matched

5. Background model • Each Lu*v* dimension is modeled with a single Gaussian • Initialize the background pixel model • No foreground objects, computing the statistics over the training sequence • Otherwise, use a bootstrapping algorithm [9] • Make use of the Mahalanobis distance to achieve segmentation, which corresponds to the probability that pixel belongs to the background [9] Gutchess et al., “A background model initialization algorithm for video surveillance”

6. Mahalanobis distance • Mahalanobis distance between each pixel and the corresponding background pixel:

7. Foreground probability model (1) • For each pixel , we have: • Feature vector • Label (0 for background, 1 for foreground) • : absolute distance of the current pixel to the background pixel in the RGB color space where , , , and are the means over the video frame in RGB space

8. Foreground probability model (2) • A color similarity measure • Computed from the cumulative histogram of all tracked objects • Computed as the number of pixels in the bin that contains divided by the number of the pixels in the color histogram • Histogram • A histogram for one component (R, G, B) describes the distribution of the number of pixels for that component color in the 16 bins

9. Foreground probability model (3) • Bayesian Network (BN) • Model the relationship of pixel label with feature vector • Model and using Gaussian mixture model (GMM) using Gaussians • The probability the a pixel belongs to the foreground is :

10. Blob Level • This paper doesn’t mention! • Foreground pixels with the same color distribution are labeled in the same class • Relabeling the disjoint blobs in connected component analysis • Adjacency • Color Similarity • Small blob

11. Components matching (1) • Find the objects in the new frame • is binarized using an adaptive threshold • A union find algorithm is used to find its 8-connected components that correspond to foreground objects • 4-connected components • 8-connected components • Foreground object matching cases: • One-to-one matching • Many-to-one matching • One-to-many matching

12. Components matching (2) • Use probabilistic matching to solve the problems mentioned above • (foreground object, connected component): • , can be described with feature vector : • : horizontal and vertical size of the bounding box • : the size in pixels • : color histogram of object/component • : centroid of all the pixels of an object/connected component • : index (i.e. or )

13. Components matching (3) • Derive information for matching a foreground object to a connected component : • : size change as • : Euclidean distance between , • : similarity between , • : horizontal size change as • : vertical size change as • If and are matched, label with , otherwise

14. Components matching (4) • Probability of matching • One-to-one matching: • Occlusion: • Object color similar to background: • Probability of foreground object disappeared is , where is dummy node

15. Object Detection • How to decide the connected components should become new object: • Define a set of feature • : size of the connected component • : distance to the nearest location of a foreground object • : a shape feature, defined as • : color similarity of object candidate to the average foreground object • Label if candidate is not new object, otherwise label

16. Experimental Results • Test a 55 minute long indoor sequence (b) Probability based only on background model (c) Probability of foreground (d)(g) Segmented objects using only background model (e)(h) Segmented objects using probability of foreground (a)(f) Foreground object

17. Conclusion • Contributions of this paper: • A new probabilistic framework for pixel segmentation and for matching of objects to blobs • A framework can account for grouping of objects • A method robust to initialization • The matching formulation is better able to model multi-object tracking and gives more reliable segmentation results