280 likes | 511 Views
A Probabilistic Framework for Video Representation . Arnaldo Mayer, Hayit Greenspan Dept. of Biomedical Engineering Faculty of Engineering Tel-Aviv University, Israel. Jacob Goldberger, CUTe Systems, Ltd. Introduction.
E N D
A Probabilistic Framework for Video Representation Arnaldo Mayer, Hayit Greenspan Dept. of Biomedical Engineering Faculty of Engineering Tel-Aviv University, Israel Jacob Goldberger, CUTe Systems, Ltd.
Introduction • In this work we describe a novel statistical video representation and modeling scheme. • Video representation schemes are needed to enable segmenting a video stream into meaningful video-objects, useful for event detection, indexing and retrieval applications.
PACS: Picture Archiving &Communication Systems Query/Retrieve Visual Information Storage Tele-Medicine Query/Retrieve Internet Database Management
What are interesting events in medical data? Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI Spatio-Temporal Tracking of Tracer in Digital Angiography
Introduction • Analysis of a video as a single entity Vs analysis of video as a sequence of frames • Inherent Spatio-temporal tracking • Gaussian Mixture Modeling in color & space-time domain t y x
Learning a Probabilistic Model in Space-Time Expectation Maximization (EM) t Gaussian Mixture Model y Feature Vectors [L,a,b,x,y,t] (6 - dimensional space) x
Video Representation via Gaussian Mixture Modeling • Each Component of the GMM Represents a Cluster in the Feature Space (=Blob) and a Spatio-temporal region in the video • PdF For the GMM : With the Parameter set
… • Given a set of feature vectors and parameter values, • the Likelihood • expresses how well the model fits the data. • The EM algorithm: iterative method to obtain the • parameter values that maximize the Likelihood
The EM Algorithm Expectation step: estimate the Gaussian clusters to which the points in feature space belong Maximization step: maximum likelihood parameter estimates using this data
Initialization & Model selection • Initialization of the EM algorithm via K-means: • Unsupervised clustering method • Non-parametric • Model selection via MDL (Minimum Description Length) • Choose k to maximize: • lk = #free parameters for a model with k mixture components
Video Model Visualization Static space-time blob Dynamic space-time blob The GMM for a given video sequence can be visualized as a set of hyper-ellipsoids (2 sigma contour) within the 6 dimensional color-space-time domain.
Detection & Recognition of Events in Video L a b x y t C Correlation coefficient : L a b x y t Cxt Cyt Ctt Ctt - Duration of space-time blob Static/Dynamic blobs - thresholds on Rxt (Hor. motion) & Ryt (Ver. motion) Direction of motion - sign of Rxt, Ryt
Detection & Recognition of Events in Video L a b x y t C Blob motion (pixels per frame) via linear regression models in space & time : L a b x y t Cxt Cyt Ctt Horizontal velocity of blob motion in image plane is extracted as the ratio of cov. parameters. Similar formalism allows for the modeling of any other motion in the image plane.
Probabilistic Image Segmentation A direct correspondence can be made between the mixture representation and the image plane. Each pixel of the original image is now affiliated with the most probable Gaussian cluster. Pixel labeling: Probability of pixel x to be labeled:
Original Model Segmentation
Original Model Segmentation Dynamic Event Tracking
Limitations of the Global Model • How can we represent non-convex spatio-temporal regions? • All the data must be available simultaneously • - Inappropriate for live video • - Model fitting time increases directly with sequence length
Piecewise Gaussian Mixture Modeling • Modeling the Video sequence • as a succession of overlapping • blocks of frames. • Obtain a succession of GMMs • instead of a single global model. • Important issues: initialization; • matching between adjacent segments for region tracking. • (“gluing”)
Piecewise GMM : “Gluing” / Matching at Junctions Frame J5 Ex: Blob matching Frame J5 blobs via GMM5 Frame J5 blobs via GMM6
Original Sequence Dynamic Event Tracking Model Sequence
Horizontal Velocity in function of Block of Frame Index Pix / frame BoF #
Vertical Velocity in function of Block of Frame Index Pix / frame BoF #
Original Sequence Segmentation Map Sequence Pix / frame Pix / frame Horizontal Velocity Vertical Velocity BOF # BOF # Sweater Trousers
Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI Sequences
Methodology Time GMM for Luminance Segmentation Maps K >= 4 1) CSF 2) White Matter 3) Gray Matter 4) Sclerotic Lesions Blobs in [L x y t] Feature Space Frame by frame Segmentation 3D (x,y,t) Connected Components
Original Sequence Dynamic Event Tracking Segmentation Maps Sequence
Time Evolution Area (in Pixels) Time point
Conclusions • The modeling and the segmentation are combined to enable the extraction of video-regions that represent coherent regions across the video sequence, otherwise termed video-objects or sub-objects. • Extracting video regions provides for a compact video content description, that may be useful for later indexing and retrieval applications. • Medical applications: lesion modeling & tracking Acknowledgment Part of the work was supported by the Israeli Ministry of Science, Grant number 05530462.