A Probabilistic Framework for Video Representation

1. A Probabilistic Framework for Video Representation Arnaldo Mayer, Hayit Greenspan Dept. of Biomedical Engineering Faculty of Engineering Tel-Aviv University, Israel Jacob Goldberger, CUTe Systems, Ltd.

2. Introduction • In this work we describe a novel statistical video representation and modeling scheme. • Video representation schemes are needed to enable segmenting a video stream into meaningful video-objects, useful for event detection, indexing and retrieval applications.

3. PACS: Picture Archiving &Communication Systems Query/Retrieve Visual Information Storage Tele-Medicine Query/Retrieve Internet Database Management

4. What are interesting events in medical data? Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI Spatio-Temporal Tracking of Tracer in Digital Angiography

5. Introduction • Analysis of a video as a single entity Vs analysis of video as a sequence of frames • Inherent Spatio-temporal tracking • Gaussian Mixture Modeling in color & space-time domain t y x

6. Learning a Probabilistic Model in Space-Time Expectation Maximization (EM) t Gaussian Mixture Model y Feature Vectors [L,a,b,x,y,t] (6 - dimensional space) x

7. Video Representation via Gaussian Mixture Modeling • Each Component of the GMM Represents a Cluster in the Feature Space (=Blob) and a Spatio-temporal region in the video • PdF For the GMM : With the Parameter set

8. … • Given a set of feature vectors and parameter values, • the Likelihood • expresses how well the model fits the data. • The EM algorithm: iterative method to obtain the • parameter values that maximize the Likelihood

9. The EM Algorithm Expectation step: estimate the Gaussian clusters to which the points in feature space belong Maximization step: maximum likelihood parameter estimates using this data

10. Initialization & Model selection • Initialization of the EM algorithm via K-means: • Unsupervised clustering method • Non-parametric • Model selection via MDL (Minimum Description Length) • Choose k to maximize: • lk = #free parameters for a model with k mixture components

11. Video Model Visualization Static space-time blob Dynamic space-time blob The GMM for a given video sequence can be visualized as a set of hyper-ellipsoids (2 sigma contour) within the 6 dimensional color-space-time domain.

12. Detection & Recognition of Events in Video L a b x y t C Correlation coefficient : L a b x y t Cxt Cyt Ctt Ctt - Duration of space-time blob Static/Dynamic blobs - thresholds on Rxt (Hor. motion) & Ryt (Ver. motion) Direction of motion - sign of Rxt, Ryt

13. Detection & Recognition of Events in Video L a b x y t C Blob motion (pixels per frame) via linear regression models in space & time : L a b x y t Cxt Cyt Ctt Horizontal velocity of blob motion in image plane is extracted as the ratio of cov. parameters. Similar formalism allows for the modeling of any other motion in the image plane.

14. Probabilistic Image Segmentation A direct correspondence can be made between the mixture representation and the image plane. Each pixel of the original image is now affiliated with the most probable Gaussian cluster. Pixel labeling: Probability of pixel x to be labeled:

15. Original Model Segmentation

16. Original Model Segmentation Dynamic Event Tracking

17. Limitations of the Global Model • How can we represent non-convex spatio-temporal regions? • All the data must be available simultaneously • - Inappropriate for live video • - Model fitting time increases directly with sequence length

18. Piecewise Gaussian Mixture Modeling • Modeling the Video sequence • as a succession of overlapping • blocks of frames. • Obtain a succession of GMMs • instead of a single global model. • Important issues: initialization; • matching between adjacent segments for region tracking. • (“gluing”)

19. Piecewise GMM : “Gluing” / Matching at Junctions Frame J5 Ex: Blob matching Frame J5 blobs via GMM5 Frame J5 blobs via GMM6

20. Original Sequence Dynamic Event Tracking Model Sequence

21. Horizontal Velocity in function of Block of Frame Index Pix / frame BoF #

22. Vertical Velocity in function of Block of Frame Index Pix / frame BoF #

23. Original Sequence Segmentation Map Sequence Pix / frame Pix / frame Horizontal Velocity Vertical Velocity BOF # BOF # Sweater Trousers

24. Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI Sequences

25. Methodology Time GMM for Luminance Segmentation Maps K >= 4 1) CSF 2) White Matter 3) Gray Matter 4) Sclerotic Lesions Blobs in [L x y t] Feature Space Frame by frame Segmentation 3D (x,y,t) Connected Components

26. Original Sequence Dynamic Event Tracking Segmentation Maps Sequence

27. Time Evolution Area (in Pixels) Time point

28. Conclusions • The modeling and the segmentation are combined to enable the extraction of video-regions that represent coherent regions across the video sequence, otherwise termed video-objects or sub-objects. • Extracting video regions provides for a compact video content description, that may be useful for later indexing and retrieval applications. • Medical applications: lesion modeling & tracking Acknowledgment Part of the work was supported by the Israeli Ministry of Science, Grant number 05530462.