190 likes | 210 Views
Segmenting Dynamic Textures with Ising Descriptors, ARX Models and Level Sets. Atiyeh Ghoreyshi and Ren é Vidal Center for Imaging Science Johns Hopkins University. Dynamic textures: modeling & recognition. A. +. z. z. v. =. 1. +. t. t. t. C. +. y. z. w. =. t. t. t.
E N D
Segmenting Dynamic Textures with Ising Descriptors, ARX Models and Level Sets Atiyeh Ghoreyshi and René Vidal Center for Imaging Science Johns Hopkins University
Dynamic textures: modeling & recognition A + z z v = 1 + t t t C + y z w = t t t dynamics images appearance • Dynamic textures are sequences of images of nonrigid scenes that exhibit certain stationarity properties in time • Can model temporal evolution of image intensities as the output of a linear dynamical system (LDS): Soatto et al. ’01
Modeling moving dynamic textures • Can we recover the rigid motion of a camera looking at a moving dynamic texture? • Can we segment a scene containing multiple dynamic textures? • Can we segment a scene containing multiple moving dynamic textures? DTCC: Dynamic Texture Constancy Constraint GPCA: Generalized Principal Component Analysis GPCA + DTCC
Dynamic textures: segmentation A + z z v A = 1 2 + t + t t z z v = 1 1 t + t t C + y z w = C 2 t t t + y z w = 1 t t t • Separate video sequence into multiple regions containing different dynamic textures (linear dynamical systems) Fixed Boundary Moving Boundary
Previous work on segmentation • Probabilistic approaches • EM with ARMA models (Chan and Vasconcelos CVPR’05) • Initialization is a problem • Does not deal with moving boundaries • Algebraic approaches • GPCA with ARMA models (Vidal and Ravichandran CVPR’05) • Spatial coherence is a problem • There have been attempts to improve the spatial coherence, but the problem still exists (Huang et al. WDV’05) • Variational approaches • Level set methods /ARMA models (Doretto et al. ICCV’03) • Cost function is an algebraic error between models • Can not deal with moving boundaries
Paper contributions • A new algorithm for the segmentation of dynamic textures • Model spatial statistics of each frame with a set of Ising descriptors • Model the dynamics of these descriptors with ARX models • Cast the segmentation problem in a variational framework that minimizes a spatial-temporal extension of the Chan-Vese energy • Key advantages of our approach • The identification of the parameters of an ARX model can be done in closed form by solving a simple linear system • Using ARX models of very low order gives good segmentation results, hence we can deal with moving boundaries more efficiently • It naturally handles intensity and texture based segmentation as well as motion based video segmentation as particular cases
Intensity-based segmentation µ ¶ @ r Z Z ³ ´ ' ' 2 2 2 2 ( ) ( ( ) ) ( ( ) ) ( ) j j ( ( ) ) ( ( ) ) ± ¸ ¸ r ¸ d d ¸ d d E C C + ¡ ¡ ¡ + ¡ + ¡ ¢ ' ¹ u x y c u x y c c c ¹ u x y c x y u x y c x y = = 1 1 2 2 1 2 1 1 2 2 ; ; j j ; ; ; ; @ r t ' ( ) ( ) C C i t n o u • Chan-Vese energy functional • Implicit methods • Represent C as the zero level set of an implicit function φ, i.e. C = {(x, y) : φ(x, y) = 0} • Do not depend on a specific parameterization of the contour • Allow topological changes of the contour during evolution • Solution • The solution to the gradient descent algorithm for φ is given by • c1 and c2 are the mean intensities inside and outside the contour C
Dynamics & intensity-based energy p F Z X j j X ( ) ( ) f f i j 1 2 2 + ¡ j j ( ( ) ( ) ) c x y a a u x y ¸ f f d d E C = = j + ¡ p 0 i ¹ u x y c x y x y ; ; ; ; ; = 1 1 ; ; ; ; X j j i 1 ( ) ( ) ( ) f f f i = + ¡ + f 1 + u x y a a u x y w x y p = = ( ) C i 0 i ; ; ; ; ; ; n i 1 F = Z X 2 ( ( ) ( ) ) ¸ f f d d + ¡ u x y c x y x y 2 2 ; ; ; ; f 1 + p = ( ) C t o u • We represent the intensities of the pixels in the images as the output of a mixture of ARX models of order p • We propose the following spatial-temporal extension of the Chan-Vese energy functionalwhere
Dynamics & intensity-based segmentation > 2 3 F 1 1 ¢ ¢ ¢ @ r ³ ' ' j j X ( ) 2 3 f I 2 ¡ ¢ ( ) ( ( ) ( ) ) ± ¸ f f r I x y j j j j + ¡ ¢ ' ¹ x y c x y ( ) ( ) 1 1 = f f 1 1 1 ; 1 ; ¡ ¡ ¢ ¢ ¢ ; ; ; ; u x y u x y j j 6 7 @ r t 1 1 k k ; ; ; ; ' j j j 6 7 f j . 1 2 1 + 6 7 p = a = = 6 7 . ; . . 4 5 . 6 7 . . F j j ( ) . . f I 4 5 x y ´ X j j j j k k ; ; 2 ( ) ( ) f f ( ( ) ( ) ) j j ¡ ¸ ¡ f f I ¡ ¡ ¢ ¢ ¢ u x y p u x y p x y c x y 1 1 k k 2 2 ; ; ; ; ; ; ; ; j j f 1 + p = • Given the ARX parameters, we can solve for the implicit function φ by solving the PDE • Given the implicit function φ, we can solve for the ARX parameters of the jth region by solving the linear system
Dynamics & texture-based segmentation ½ 8 j ( ) ( ) j f f f i 1 ¡ P ( ) ¡ < f f 6 ¢ i i 5 r s u u ² x y = ( ) ; ; f ¢ c ; ; > x y = < c ; ; h i 1 C t + 2 c o e r w s e ( ) i f u x y = i 1 ; ; P ( ) f f i i 5 u x y = 2 ; ; > w : ( ) W 2 x y ; • We use the Ising second order model to represent the static texture at pixel (x,y) and frame f • Consider the set of all cliques as shown below in a neighborhood W of size w×w around the pixel: • For each clique, one defines the function • The ith entry of the texture descriptor is defined as
Dynamics & texture-based segmentation p p F X X j j j j Z ( ( ) ) ( ( ) ) ( ) f f f f f A A A i i j 1 2 + + ¡ ¡ + X c u a u u w x x y y x x y y x y = = = j 2 0 0 i i j j k ( ) ( ) k ; ; ; ; ¸ ; ; ; ; f ; ; f ; d d E C + ¡ u c ¹ x y x y x y = 1 1 ; ; ; ; i i 1 1 = = f 1 + p = ( ) C i n F Z X 2 k ( ) ( ) k ¸ f f d d + ¡ u c x y x y x y 2 2 ; ; ; ; f 1 + p = ( ) C t o u • We model the temporal evolution of the texture descriptors using a mixture of vector-valued ARX models of the form • We segment the scene by minimizing the energy functionalwhere
Dynamics & texture-based segmentation • Given the ARX parameters, we can solve for the implicit function φ by solving a PDE • Given the implicit function φ, we can solve for the ARX parameters of the jth region by solving a linear system • Implementation: we assume that the ARX parameter matrices are diagonal, which implies that the entries of the feature vector are the outputs of five decoupled scalar ARX models. Therefore, we can estimate the ARX parameters independently for each entry of u, as we did before for the intensities.
Dynamic textures with moving boundaries • In order to follow the moving boundaries, we use the method described for fixed boundaries on a temporal window of frames as follows: • Given a user specified embedding φ0 representing an initial contour, apply the dynamic texture segmentation method to frames 1,…,F. This results in a new embedding, φ1 • Use φ1 as an initial embedding, and apply the dynamic texture segmentation method to frames 2,…,F+1. This results in a new embedding φ2 • Repeat the previous step for all the remaining frames of the sequence
Experimental results • Fixed boundary segmentation results and comparison Original sequence ARMA GPCAVidal CVPR’05 ARMASubspace Angles Level Sets Doretto ICCV’03 Our method
Experimental results • Fixed boundary segmentation results and comparison Ocean-dynamics Ocean-appearance
Experimental results • Moving boundary segmentation results and comparison Ocean-fire sequence
Experimental results • Results on a real sequence Raccoon on river sequence
Conclusions • A new algorithm for the segmentation of dynamic textures • Model spatial statistics of each frame with a set of Ising descriptors • Model the dynamics of these descriptors with ARX models • Cast the segmentation problem in a variational framework that minimizes a spatial-temporal extension of the Chan-Vese energy • Key advantages of our approach • The identification of the parameters of an ARX model can be done in closed form by solving a simple linear system • Using ARX models of very low order gives good segmentation results, hence we can deal with moving boundaries more efficiently • It naturally handles intensity and texture based segmentation as well as motion based video segmentation as particular cases
Experimental Results • Segmentation of the epicardium One axial slice of a beating heart Segmentation results