Some Rambling (Müjdat) and Some More Serious Discussion (Ayres, Junmo, Walter) on Shape Priors

1. Some Rambling (Müjdat)and Some More Serious Discussion (Ayres, Junmo, Walter)on Shape Priors

2. Segmenting curve Observed image data Desire to use Shape Priors in Segmentation • The posterior: • The most common (implicit) prior used is the curve length penalty: • Want to be able to use better prior models

3. Challenges and remarks • Need probabilistic descriptions in the space of shapes • A non-linear, infinite-dimensional manifold • Distance (similarity) measures in the shape space • Of course, a statistical description for shapes has uses other than segmentation as well • Sampling from a shape density • Recognition of objects • Completion of incomplete shapes

4. The PCA World • Cootes & Taylor • Shape representation using marker points • Leventon, Tsai • PCA and level sets • More… • Want a more principled approach

5. Some Literature • “Active shape models - their training and application,” T. Cootes, C. J. Taylor, D.H. Cooper, J. Graham, 1995. • “Embedding Gestalt laws in Markov random fields,” Song-Chun Zhu, 1999. • “On the incorporation of shape into geometric active contours,” Y. Chen, H.D. Tagare et al., 2001. • “Image segmentation based on prior probabilistic shape models,” A. Litvin and W. C. Karl, 2002. • “Shape priors for level set representations,” M. Rousson and N. Paragios, ECCV, 2002. • “Nonlinear shape statistics in Mumford-Shah based segmentation,” D. Cremers, T. Kohlberger, and C. Schnorr, 2002. • “Geometric analysis of constrained curves for image understanding,” A. Srivastava, W. Mio, E. Klassen, X. Liu, 2003. • “Analysis of planar shapes using geodesic paths on shape spaces,” E. Klassen, A. Srivastava, and W. Mio (in review) • “Gaussian distributions on Lie groups and their application to stat. shape analysis,” P. T. Fletcher, S. Joshi, C. Lu, and S. Pizer, 2003.

6. Overview of Anuj Srivastava’s Work • Specify a space of continuous curves with constraints (e.g. simple closed) and exploit the differential geometry of this space • Use geodesic paths for deformations between curves and to compute distances – do not have analytical exps. for geodesics • To move in the shape manifold, first move in a linear space and then project back (using tangents/normals) • Given two curves, solve an optimization problem to find the geodesic path between them (find a local min) • Build statistical descriptions based on Karcher means; covariance of (Fourier coefficients of) tangent vectors • Use such descriptions as priors (very preliminary) • Some current limitations: • Cannot handle topological changes • Extension to surfaces in 3-D not straightforward

7. Outline • Some metrics proposed for shape similarity (Junmo) • Anuj Srivastava’s work • Geometric representations of curves and shape spaces (Walter) • Tangents, normals, geodesics (Ayres) • Statistical models and application to segmentation (Junmo) • Brief highlights from Pizer’s work (Walter) • Discussion on all of this

8. Some Metrics on the Space of Shapes • Notion of “similarity” between shapes • Basic task of vision system is to recognize similar objects which belong to the same category. • Describing shapes : landmarks, level set • Shape can be described as • There are several metrics for two shapes

9. Hausdorff Metric • Given • Very sensitive to any outlier points in and

10. Template Metric • Totally insensitive to outliers

11. Transport Metric • Fill with ‘stuff’ and find the shortest paths along which to move this ‘stuff’ so that it now fills

12. Optimal Diffeomorphism • If and are topologically different, the distance would be infinite. • E.g. is minus a pinhole • E.g. A small break cuts a shape into two

13. Geometric Representations of Curves and Shape Spaces • Restrict attention to closed curves in R2. • Classify curves which differ only by orientation preserving rigid motions (rotation and translation) and uniform scaling as the same shape. • Consider two different representations of planar curves for simple closed curves: • Using direction functions. • Using curvature functions.

14. Preliminaries • First, the issue of scaling is resolved by fixing the length of all curves to 2. • Curves are parameterized by arc length with period 2 & • Define the unit tangent function where S1 is the unit circle, & where (s) is the direction function. • (s+2) - (s) = 2n (n = rotation index { = 1})

15. Curves to Consider • Consider entire set of curves with rotation index 1 because this set is complete. • Contains its own limit points. • Note that the set of simple closed curves is an open subset of this larger set.

16. Shape Using Direction Functions • Direction function for S1 is 0(s) = s. All other closed curves have direction function  = 0 + f, where f is L2periodic on [0,2]. • To adjust for rigid rotations, restrict attention to  s.t. • To ensure curve closure, require

17. More Direction Functions • Define a map by • The pre-shape space C1 is 1-1(,0,0). • Multiple elements of C1 may denote the same shape. An adjustment of the reference point (s=0) handles this.

18. Geometry of Shape Manifolds • Constraints define a manifold embedded in q0 + L2 • Move along manifold by moving in tangent space and projecting back to manifold • Tangent space is infinite dimensional, but normal space is characterized by three constraints defined in f1

19. Tangents and Normals • The derivative of f1 in the direction of f at  is: • Implies df1 is surjective • If f is orthogonal to {1, sin q, cos q}, then df1=0 in the direction of f and hence f is in the tangent space

20. Projections • Want to find the closest element in C1 to an arbitrary qq0 + L2 • Basic idea: move orthogonal to level sets so projections under f form a straight line in R3 • For a point b  R3, we define the level set as: • Let b1=(p,0,0). Then its level set is the preshape space C1

21. Approximate Projections • If points are close to C1, then one can use a faster method • Let dq be the normal vector at q for which f(q+dq)=b1. Can do first order approximation to compute this • Approximate Jacobian as:

22. Iterative algorithm • Define the residual (error) vector as • Then: where • Iteratively update q + dqq until the error goes to zero • Call this projection operator P

23. Example Projections Fig. 1: Projections of arbitrary curves into C1

24. Geodesics • Definition: For a manifold embedded in Euclidean space, a geodesic is a constant speed curve whose acceleration vector is always perpendicular to the manifold • Define the metric between two shapes as the distance along the manifold between the shapes with respect to the L2 inner product • Nice features: • Defined for all closed curves • Interpolants are closed curves • Finds geodesics in a local sense, not necessarily global

25. Paths from initial conditions • Assume we have a q in C1 and an f in the tangent space • Approximate geodesic along manifold by moving to q+fDt and projecting that back onto the manifold (Dt is step size) • So q(t+Dt) = P(q(t)+f(t)Dt)

26. Transporting the tangent vector • Now f(t) is not in the tangent space of q(t+Dt) • Two conditions for a geodesic: • The acceleration vector must be perpendicular to the manifold: simply project f into the next tangent space • The curve must move at constant speed: renormalize so ||f(t+1)||=||f(t)|| • hk is the orthonormal basis of the normal space

27. Geodesics on shape spaces • S1 is a quotient space of C1 under actions of S1 by isometries, so finding geodesics in S1 equivalent to finding geodesics in C1 which are orthogonal to S1 orbits • S1 acting by isometries implies that if a geodesic in preshape space is orthogonal to one S1 orbit, it’s orthogonal to all S1 orbits which it meets • So now normal space has one additional component spanned by • The algorithm is the same as detailed earlier except with an expanded normal space

28. Geodesics between shapes • We know how to generate geodesic paths given q and f • Now we want to construct a geodesic path from q1 to q2 • So we need to find all f that lead from q1 to an S1 orbit of q2 in unit time, and then choose the one that leads to the shortest path • Let Y define the geodesic flow, with (q1,0,f)=q1 as the initial condition • We then want Y(q1,1,f)=q2

29. Finding the geodesic • Define an error functional which measures how close we are to the target at t=1: • Choose the geodesic as the flow Y which has the smallest initial velocity ||f|| • i.e., min ||f|| s.t. H[f]=0 • Hard because infinite dimensional search

30. Fourier decomposition • f  L2, so it has a Fourier decomposition • Approximate f with its first m+1 cosine components and its first m sine components: • Let a be the vector containing all of the Fourier coefficients • Now optimization problem is min ||a|| s.t. H[a]=0

31. Geodesic paths Fig. 2: Geodesic paths between two shapes

32. Statistical Modeling of Shapes • Given example shapes • Mean shape • Shape variation • Shape prior • Sampling from the prior • Using shape prior for segmentation of occluded images

33. Mean Shape • Given the geodesic distance function , • Karcher mean of shapes is defined to be a shape for which the variance function is a local minimum • The Karcher mean exists, but may not be unique

34. Variances on Shape Spaces • Model the variation from the mean shape as , an element of the tangent space at the mean shape • Represent by its Fourier expansion: • Model as multivariate normal with mean 0 and covariance matrix

35. Shape Sampling • Sample Fourier coefficients of tangent vectors from the multivariate normal distribution • Move along the geodesic path starting from the mean shape in the direction of by distance

36. Shape Sampling Examples Random samples from the Gaussian model Mean shape Observed shapes

37. Shape Prior • : the space of curves (larger than shape space) • can be represented as pairs • : parameters for translation, rotation, and scaling • : the shape • Gaussian density with a mean shape with the shape dispersion

38. Bayesian Discovery of Objects

39. Some closing thoughts on Anuj Srivastava’s work • Most energy has been spent on manipulating the shape manifold • Work on using these models as priors preliminary • Non-diagonal covariance – explore modes of variation • Non-Gaussian? • Mean in manifold? • Other thoughts • Non-Fourier representations for tangents – KL expansion? • Could similar ideas be used with representations based on signed distance functions?

40. Overview of Steve Pizer’s Work • Medial representations (m-reps) are used to model the geometry of anatomical objects. • Medial parameters are not in a Euclidean space; so, PCA cannot be used. However, m-reps model parameters are elements of a Lie group. • Gaussian distributions on this Lie group are considered, with the max likelihood estimates of mean and covariance derived. • Similar to PCA for Euclidean spaces, principal geodesic analysis (PGA) on Lie groups are defined for the study of anatomical variability. • Framework is applied to hippocampi in a schizophrenia study. • 86 m-rep figures are first aligned (translation/rotation/scaling) • Intrinsic mean is then computed • PGA (modes of variability) are then computed • Results yield smoother deformations when compared with PCA

41. Medial Representation • Introduced by Blum (1978), a 3-D object is represented by a set of connected continuous medial manifolds formed by the centers of all spheres are are interior to the object and tangent to the object boundary at two or more points. The figure below illustrates this:

42. Medial Atom & Lie Groups • A medial atom is represented by • The location in space (R3) • The radius of the sphere (R+) • The local frame (SO(3)) • The object angle (SO(2)) • R3 is a Lie group under vector addition, R+ is a multiplicative Lie group, and SO(2) & SO(3) are Lie groups under composition of rotations. • The direct product of Lie groups is a Lie group.

43. Lie Groups and Lie Algebras • A Lie group is a group G that is a finite-dim manifold such that the two group operations of G, multiplication and inverse, are C2 mappings. • If e is the identity of G, the tangent space at e forms a Lie algebra. The exponential map provides a method for mapping vectors in the tangent space into the Lie group.

44. Alignment and PGA • Translation: each model is situated so that the average of its medial atoms is at the origin • Rotation and scaling are done in a manner which minimizes the total sum-of-squared distances between m-rep figures. • After alignment, principal directions in the geodesic are computed, and the analog to PCA is performed.

45. Results • The analysis, performed on 86 aligned hippocampus m-reps, shows smooth deformations (compared with PCA). The mean shape is top left, the medial atoms are overlaid lower left, and the first 3 PGA modes are shown right.

46. Shape Using Curvature Functions • Alternatively, curves can be represented by curvature fcns. Since the rotation index is 1, • Using , the closure condition

47. More Curvature Functions • Define a map by then the pre-shape space C2 is 2-1(2,0,0). • As with direction functions, different placements of s=0 result in differentC2 shapes. Thus, re-parameterization is needed.