Segmenting Dynamic Textures with Ising Descriptors, ARX Models and Level Sets

1. Segmenting Dynamic Textures with Ising Descriptors, ARX Models and Level Sets Atiyeh Ghoreyshi and René Vidal Center for Imaging Science Johns Hopkins University

2. Dynamic textures: modeling & recognition A + z z v = 1 + t t t C + y z w = t t t dynamics images appearance • Dynamic textures are sequences of images of nonrigid scenes that exhibit certain stationarity properties in time • Can model temporal evolution of image intensities as the output of a linear dynamical system (LDS): Soatto et al. ’01

3. Modeling moving dynamic textures • Can we recover the rigid motion of a camera looking at a moving dynamic texture? • Can we segment a scene containing multiple dynamic textures? • Can we segment a scene containing multiple moving dynamic textures? DTCC: Dynamic Texture Constancy Constraint GPCA: Generalized Principal Component Analysis GPCA + DTCC

4. Dynamic textures: segmentation A + z z v A = 1 2 + t + t t z z v = 1 1 t + t t C + y z w = C 2 t t t + y z w = 1 t t t • Separate video sequence into multiple regions containing different dynamic textures (linear dynamical systems) Fixed Boundary Moving Boundary

5. Previous work on segmentation • Probabilistic approaches • EM with ARMA models (Chan and Vasconcelos CVPR’05) • Initialization is a problem • Does not deal with moving boundaries • Algebraic approaches • GPCA with ARMA models (Vidal and Ravichandran CVPR’05) • Spatial coherence is a problem • There have been attempts to improve the spatial coherence, but the problem still exists (Huang et al. WDV’05) • Variational approaches • Level set methods /ARMA models (Doretto et al. ICCV’03) • Cost function is an algebraic error between models • Can not deal with moving boundaries

6. Paper contributions • A new algorithm for the segmentation of dynamic textures • Model spatial statistics of each frame with a set of Ising descriptors • Model the dynamics of these descriptors with ARX models • Cast the segmentation problem in a variational framework that minimizes a spatial-temporal extension of the Chan-Vese energy • Key advantages of our approach • The identification of the parameters of an ARX model can be done in closed form by solving a simple linear system • Using ARX models of very low order gives good segmentation results, hence we can deal with moving boundaries more efficiently • It naturally handles intensity and texture based segmentation as well as motion based video segmentation as particular cases

7. Intensity-based segmentation µ ¶ @ r Z Z ³ ´ ' ' 2 2 2 2 ( ) ( ( ) ) ( ( ) ) ( ) j j ( ( ) ) ( ( ) ) ± ¸ ¸ r ¸ d d ¸ d d E C C + ¡ ¡ ¡ + ¡ + ¡ ¢ ' ¹ u x y c u x y c c c ¹ u x y c x y u x y c x y = = 1 1 2 2 1 2 1 1 2 2 ; ; j j ; ; ; ; @ r t ' ( ) ( ) C C i t n o u • Chan-Vese energy functional • Implicit methods • Represent C as the zero level set of an implicit function φ, i.e. C = {(x, y) : φ(x, y) = 0} • Do not depend on a specific parameterization of the contour • Allow topological changes of the contour during evolution • Solution • The solution to the gradient descent algorithm for φ is given by • c1 and c2 are the mean intensities inside and outside the contour C

8. Dynamics & intensity-based energy p F Z X j j X ( ) ( ) f f i j 1 2 2 + ¡ j j ( ( ) ( ) ) c x y a a u x y ¸ f f d d E C = = j + ¡ p 0 i ¹ u x y c x y x y ; ; ; ; ; = 1 1 ; ; ; ; X j j i 1 ( ) ( ) ( ) f f f i = + ¡ + f 1 + u x y a a u x y w x y p = = ( ) C i 0 i ; ; ; ; ; ; n i 1 F = Z X 2 ( ( ) ( ) ) ¸ f f d d + ¡ u x y c x y x y 2 2 ; ; ; ; f 1 + p = ( ) C t o u • We represent the intensities of the pixels in the images as the output of a mixture of ARX models of order p • We propose the following spatial-temporal extension of the Chan-Vese energy functionalwhere

9. Dynamics & intensity-based segmentation > 2 3 F 1 1 ¢ ¢ ¢ @ r ³ ' ' j j X ( ) 2 3 f I 2 ¡ ¢ ( ) ( ( ) ( ) ) ± ¸ f f r I x y j j j j + ¡ ¢ ' ¹ x y c x y ( ) ( ) 1 1 = f f 1 1 1 ; 1 ; ¡ ¡ ¢ ¢ ¢ ; ; ; ; u x y u x y j j 6 7 @ r t 1 1 k k ; ; ; ; ' j j j 6 7 f j . 1 2 1 + 6 7 p = a = = 6 7 . ; . . 4 5 . 6 7 . . F j j ( ) . . f I 4 5 x y ´ X j j j j k k ; ; 2 ( ) ( ) f f ( ( ) ( ) ) j j ¡ ¸ ¡ f f I ¡ ¡ ¢ ¢ ¢ u x y p u x y p x y c x y 1 1 k k 2 2 ; ; ; ; ; ; ; ; j j f 1 + p = • Given the ARX parameters, we can solve for the implicit function φ by solving the PDE • Given the implicit function φ, we can solve for the ARX parameters of the jth region by solving the linear system

10. Dynamics & texture-based segmentation ½ 8 j ( ) ( ) j f f f i 1 ¡ P ( ) ¡ < f f 6 ¢ i i 5 r s u u ² x y = ( ) ; ; f ¢ c ; ; > x y = < c ; ; h i 1 C t + 2 c o e r w s e ( ) i f u x y = i 1 ; ; P ( ) f f i i 5 u x y = 2 ; ; > w : ( ) W 2 x y ; • We use the Ising second order model to represent the static texture at pixel (x,y) and frame f • Consider the set of all cliques as shown below in a neighborhood W of size w×w around the pixel: • For each clique, one defines the function • The ith entry of the texture descriptor is defined as

11. Dynamics & texture-based segmentation p p F X X j j j j Z ( ( ) ) ( ( ) ) ( ) f f f f f A A A i i j 1 2 + + ¡ ¡ + X c u a u u w x x y y x x y y x y = = = j 2 0 0 i i j j k ( ) ( ) k ; ; ; ; ¸ ; ; ; ; f ; ; f ; d d E C + ¡ u c ¹ x y x y x y = 1 1 ; ; ; ; i i 1 1 = = f 1 + p = ( ) C i n F Z X 2 k ( ) ( ) k ¸ f f d d + ¡ u c x y x y x y 2 2 ; ; ; ; f 1 + p = ( ) C t o u • We model the temporal evolution of the texture descriptors using a mixture of vector-valued ARX models of the form • We segment the scene by minimizing the energy functionalwhere

12. Dynamics & texture-based segmentation • Given the ARX parameters, we can solve for the implicit function φ by solving a PDE • Given the implicit function φ, we can solve for the ARX parameters of the jth region by solving a linear system • Implementation: we assume that the ARX parameter matrices are diagonal, which implies that the entries of the feature vector are the outputs of five decoupled scalar ARX models. Therefore, we can estimate the ARX parameters independently for each entry of u, as we did before for the intensities.

13. Dynamic textures with moving boundaries • In order to follow the moving boundaries, we use the method described for fixed boundaries on a temporal window of frames as follows: • Given a user specified embedding φ0 representing an initial contour, apply the dynamic texture segmentation method to frames 1,…,F. This results in a new embedding, φ1 • Use φ1 as an initial embedding, and apply the dynamic texture segmentation method to frames 2,…,F+1. This results in a new embedding φ2 • Repeat the previous step for all the remaining frames of the sequence

14. Experimental results • Fixed boundary segmentation results and comparison Original sequence ARMA GPCAVidal CVPR’05 ARMASubspace Angles Level Sets Doretto ICCV’03 Our method

15. Experimental results • Fixed boundary segmentation results and comparison Ocean-dynamics Ocean-appearance

16. Experimental results • Moving boundary segmentation results and comparison Ocean-fire sequence

17. Experimental results • Results on a real sequence Raccoon on river sequence

18. Conclusions • A new algorithm for the segmentation of dynamic textures • Model spatial statistics of each frame with a set of Ising descriptors • Model the dynamics of these descriptors with ARX models • Cast the segmentation problem in a variational framework that minimizes a spatial-temporal extension of the Chan-Vese energy • Key advantages of our approach • The identification of the parameters of an ARX model can be done in closed form by solving a simple linear system • Using ARX models of very low order gives good segmentation results, hence we can deal with moving boundaries more efficiently • It naturally handles intensity and texture based segmentation as well as motion based video segmentation as particular cases

19. Experimental Results • Segmentation of the epicardium One axial slice of a beating heart Segmentation results