3D Rigid/Nonrigid Registration

1. 3D Rigid/Nonrigid Registration • Known features, correspondences, transformation model – feature based • Specific motion type, unknown correspondences – feature based • 3) Known transformation model, unknown correspondences – region based • 4) Specific motion model – feature based • 5) Unknown motion model, unknown correspondences – region based

2. Visual Motion Jim Rehg (G.Tech)

3. Motion (Displacement) of Environment Image plane Scene Flow Motion Field Visual motion results from the displacement of the scene with respect to a fixed camera (or vice-versa). Motion field is the 2-D velocity field that results from a projection of the 3-D scene velocities

4. Examples of Visual Motion

5. Examples of Visual Motion

6. Examples of Visual Motion

7. Applications of Motion Analysis • Visual tracking • Structure recovery • Robot (vehicle) navigation

8. Applications of Motion Analysis • Visual tracking • Structure recovery • Robot (vehicle) navigation

9. Motion Segmentation • Where are the independently moving objects (and how many are there)?

10. Optical Flow • 2-D velocity field describing the apparent motion in an image sequence • A vector at each pixel indicates its motion (between a pair of frames). Horn and Schunk Ground truth

11. Optical Flow and Motion Field • In general the optical flow is an approximation to the motion field. • When the scene can be segmented into rigidly moving objects (for example) the relationship between the two can be made precise. • We can always think of the optical flow as summarizing the temporal change in an image sequence.

12. Computing Optical Flow Courtesy of Michael Black

13. Cost Function for Optical Flow Courtesy of Michael Black

14. Lucas-Kanade Method • Brute-force minimization of SSD error can be inefficient and inaccurate • Many redundant window evaluations • Answer is limited to discrete u, v pairs

15. Lucas-Kanade Method • Problems with brute-force minimization of SSD error • Many redundant window evaluations • Answer is limited to discrete u, v pairs • Related to Horn-Schunk optical flow equations • Several key innovations • Early, successful use of patch-based model in low-level vision. Today these models are used everywhere. • Formulation of vision problem as non-linear least squares optimization, a trend which continues to this day.

16. Optical Flow Estimation

17. Optical Flow Estimation

18. Optical Flow Constraint

19. Optimization

20. Optimization

21. Optimization

22. Quality of Image Patch • Eigenvalues of the matrix contain information about local image structure • Both eigenvalues (close to) zero: Uniform area • One eigenvalue (close to) zero: Edge • No eigenvalues (close to) zero: Corner

23. Contributions of Lucas-Kanade • Basic idea of patch or template is very old (goes back at least to Widrow) • But in practice patch models have worked much better than the alternatives: • Point-wise differential equations with smoothness • Edge-based descriptions • Patchs provide a simple compact enforcement of spatial continuity and support (robust) least-squares estimators.

24. Lets Talk Applications

25. Summer School 2005 Medical Image Registration (a short overview) Alain Pitiot, Ph.D. Siemens Molecular Imaging - Advanced Applications

26. SOME APPLICATIONS • Medical Image Registration

27. Motivation • Advances in imaging technology  novel modalities • see beyond: inside (non-invasive), • during (dynamic processes), • at small scale (increased resolution) • Understanding and correlating structure & function • - automated/aided diagnosis • - image guided surgery/radio-therapy • - treatment/surgery planning • - medical atlases • - longitudinal studies: disease progression, development

28. Definitions • Def. #1:put two images into spatial correspondence • goal: extract more/better information • Def. #2: maximize similarity between transformed source image & target image source image target image + transformed target image CT (thorax) PET (thorax) Anatomical Functional

29. Taxonomy • Nature of application • Subject • - intrasubject • - intersubject • - atlas Homer Simpson (MRI, coronal section)

30. Taxonomy • Nature of application • Subject • - intrasubject • - intersubject • - atlas Homer Simpson (monkey position) Homer Simpson (rest position)  very similar shapes

31. Taxonomy • Nature of application • Subject • - intrasubject • - intersubject • - atlas Homo sapiens sapiens brain Homer Simpson  expect larger differences

32. Taxonomy • Nature of application • Subject • - intrasubject • - intersubject • - atlas Homer Simpson (MRI) Homer Simpson (labelled atlas)

33. Taxonomy • Nature of application • Subject • - intrasubject • - intersubject • - atlas • Registration basis • - extrinsic • - intrinsic

34. Taxonomy • Nature of application • Subject • - intrasubject • - intersubject • - atlas • Registration basis • - extrinsic • - intrinsic fast, explicit computation prospective, often invasive, often rigid transf. only stereotactic frame

35. Taxonomy • Nature of application • Subject • - intrasubject • - intersubject • - atlas • Registration basis • - extrinsic • - intrinsic versatile, minimally invasive no ground truth PET scintillography

36. Taxonomy | Nature of Application • Registration basis extrinsic intrinsic • - landmark based • - segmentation based • - voxel based CT PET fast accuracy limited by localization precision

37. Taxonomy | Nature of Application • Registration basis extrinsic intrinsic • - landmark based • - segmentation based • - voxel based segmented corpora callosa fast accuracy limited by segmentation  combine with voxel based

38. Taxonomy | Nature of Application • Registration basis extrinsic intrinsic • - landmark based • - segmentation based • - voxel based cryo. section myelin-stained histological section most flexible approach resource intensive  combine with previous techniques

39. Taxonomy • Nature of input images • Modality • Combination: • - mono-modal: same modality for source and target • - multi-modal: different modality • Dimensionality • - spatial: 2-D/2-D, 2-D/3-D, 3-D/3-D • - temporal a few imaging modalities

40. Taxonomy global local • Constraints • fusion  maximize similarity between • transformed source & target • Transformation space • - flexibility • rigid, affine, parameterized, • free-form • - support • local, global rigid affine parameterized fluid/elastic  choose space that fits anatomy and/or application

41. Taxonomy • Constraints • Similarity measure • “intensities of matched images • verify criterion” • - to each hypothesis its measure: conservation affine functional statistical • conservation of intensity  SSD • affine relationship  correlation coefficient • functional relationship  correlation ratio • statistical dependence  mutual information source target

42. Taxonomy • Constraints • Similarity measure • “intensities of matched images • verify criterion” • - to each hypothesis its measure: conservation affine functional statistical • conservation of intensity  SSD • affine relationship  correlation coefficient • functional relationship  correlation ratio • statistical dependence  mutual information source target

43. Taxonomy • Constraints • Similarity measure • “intensities of matched images • verify criterion” • - to each hypothesis its measure: conservation affine functional statistical • conservation of intensity  SSD • affine relationship  correlation coefficient • functional relationship  correlation ratio • statistical dependence  mutual information source target

44. Taxonomy • Constraints • Similarity measure • “intensities of matched images • verify criterion” • - to each hypothesis its measure: conservation affine functional statistical • conservation of intensity  SSD • affine relationship  correlation coefficient • functional relationship  correlation ratio • statistical dependence  mutual information source target

45. Taxonomy • Constraints • Similarity measure • “intensities of matched images • verify criterion” • - to each hypothesis its measure: conservation affine functional statistical • conservation of intensity  SSD • affine relationship  correlation coefficient • functional relationship  correlation ratio • statistical dependence  mutual information source target

46. Taxonomy • Optimization • Often iterative • - deterministic • gradient descent • - stochastic • simulated annealing

47. Taxonomy • Optimization • Often iterative • - deterministic • gradient descent • - stochastic • simulated annealing

48. Taxonomy • Optimization • Often iterative • - deterministic • gradient descent • - stochastic • simulated annealing

49. Taxonomy • Optimization • Often iterative • - deterministic • gradient descent • - stochastic • simulated annealing • Heteroclite bag of tricks • - progressive refinement • - multi-scale (multi-resolution)

50. Taxonomy • Optimization • Often iterative • - deterministic • gradient descent • - stochastic • simulated annealing • bag of tricks • - progressive refinement • - multi-scale (multi-resolution)