1 / 96

Iterative Image Registration: Lucas & Kanade Revisited

Iterative Image Registration: Lucas & Kanade Revisited. Kentaro Toyama Vision Technology Group Microsoft Research. Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges.

trey
Download Presentation

Iterative Image Registration: Lucas & Kanade Revisited

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Iterative Image Registration:Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group Microsoft Research

  2. Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges

  3. Bergen, Anandan, Hanna, Hingorani (ECCV 1992) • Shi & Tomasi (CVPR 1994) • Szeliski & Coughlan (CVPR 1994) • Szeliski (WACV 1994) • Black & Jepson (ECCV 1996) • Hager & Belhumeur (CVPR 1996) • Bainbridge-Smith & Lane (IVC 1997) • Gleicher (CVPR 1997) • Sclaroff & Isidoro (ICCV 1998) • Cootes, Edwards, & Taylor (ECCV 1998) SC G SI CET BAHH ST S BJ HB BL History • Lucas & Kanade (IUW 1981) LK

  4. Image Registration

  5. Applications

  6. SC G SI CET BAHH ST S BJ HB BL LK Applications • Stereo

  7. Applications • Stereo • Dense optic flow SC G SI CET BAHH ST S BJ HB BL LK

  8. Applications • Stereo • Dense optic flow • Image mosaics SC G SI CET BAHH ST S BJ HB BL LK

  9. Applications • Stereo • Dense optic flow • Image mosaics • Tracking SC G SI CET BAHH ST S BJ HB BL LK

  10. Applications • Stereo • Dense optic flow • Image mosaics • Tracking • Recognition ? SC G SI CET BAHH ST S BJ HB BL LK

  11. Derivation #1 Lucas & Kanade

  12. I0(x) L&K Derivation 1

  13. h I0(x) I0(x+h) L&K Derivation 1

  14. L&K Derivation 1 h I0(x) I(x)

  15. L&K Derivation 1 h I0(x) I(x)

  16. L&K Derivation 1 I0(x) I(x) R

  17. L&K Derivation 1 I0(x) I(x)

  18. L&K Derivation 1 h0 I0(x) I(x)

  19. L&K Derivation 1 I0(x+h0) I(x)

  20. L&K Derivation 1 I0(x+h1) I(x)

  21. L&K Derivation 1 I0(x+hk) I(x)

  22. L&K Derivation 1 I0(x+hf) I(x)

  23. Lucas & KanadeDerivation #2

  24. E(h) = S [ I(x) - I0(x+h) ]2 E(h) S [ I(x) - I0(x) - hI0’(x) ]2 xeR xeR L&K Derivation 2 • Sum-of-squared-difference (SSD) error

  25. SI0’(x)(I(x) - I0(x)) xeR h SI0’(x)2 xeR L&K Derivation 2 S 2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] xeR =0

  26. w(x)[I(x) - I0(x)] S I0’(x) x h Sw(x) x SI0’(x)[I(x) - I0(x)] x h SI0’(x)2 x Comparison

  27. Comparison w(x)[I(x) - I0(x)] S I0’(x) x h Sw(x) x SI0’(x)[I(x) - I0(x)] x h SI0’(x)2 x

  28. Generalizations

  29. Original S [ ] ( I ( x h ) - I0 ( x ) E h ) = + 2 x e R

  30. SC G SI CET BAHH ST S BJ HB BL LK Original • Dimension of image S [ ] ( I ( x h ) - I0 ( x ) E h ) = + 2 x e R 1-dimensional

  31. SC G SI CET BAHH ST S BJ HB BL LK Generalization 1a • Dimension of image S [ ] ( I ( x h ) - I0 ( x ) E h ) = + 2 x e R 2D:

  32. Generalization 1b • Dimension of image S [ ] ( I ( x h ) - I0 ( x ) E h ) = + 2 x e R Homogeneous 2D: SC G SI CET BAHH ST S BJ HB BL LK

  33. SC G SI CET BAHH ST S BJ HB BL LK Problem A Does the iteration converge?

  34. Problem A Local minima:

  35. Problem A Local minima:

  36. h is undefined if SI0’(x)2 is zero xeR Problem B Zero gradient: -SI0’(x)(I(x) - I0(x)) h xeR SI0’(x)2 xeR SC G SI CET BAHH ST S BJ HB BL LK

  37. ? Problem B Zero gradient:

  38. Problem B’ Aperture problem: -S (x)(I(x) - I0(x)) xeR hy S 2 xeR SC G SI CET BAHH ST S BJ HB BL LK

  39. ? Problem B’ No gradient along one direction:

  40. Solutions to A & B • Possible solutions: • Manual intervention SC G SI CET BAHH ST S BJ HB BL LK

  41. Solutions to A & B • Possible solutions: • Manual intervention • Zero motion default SC G SI CET BAHH ST S BJ HB BL LK

  42. Solutions to A & B • Possible solutions: • Manual intervention • Zero motion default • Coefficient “dampening” SC G SI CET BAHH ST S BJ HB BL LK

  43. Solutions to A & B • Possible solutions: • Manual intervention • Zero motion default • Coefficient “dampening” • Reliance on good features SC G SI CET BAHH ST S BJ HB BL LK

  44. Solutions to A & B • Possible solutions: • Manual intervention • Zero motion default • Coefficient “dampening” • Reliance on good features • Temporal filtering SC G SI CET BAHH ST S BJ HB BL LK

  45. Solutions to A & B • Possible solutions: • Manual intervention • Zero motion default • Coefficient “dampening” • Reliance on good features • Temporal filtering • Spatial interpolation / hierarchical estimation SC G SI CET BAHH ST S BJ HB BL LK

  46. Solutions to A & B • Possible solutions: • Manual intervention • Zero motion default • Coefficient “dampening” • Reliance on good features • Temporal filtering • Spatial interpolation / hierarchical estimation • Higher-order terms SC G SI CET BAHH ST S BJ HB BL LK

  47. Original S [ ] ( I ( x h ) - I0 ( x ) E h ) = + 2 x e R

  48. Original • Transformations/warping of image S [ ] ( I ( x h ) - I0 ( x ) E h ) = + 2 x e R Translations: SC G SI CET BAHH ST S BJ HB BL LK

  49. Problem C What about other types of motion?

  50. Generalization 2a • Transformations/warping of image S [ ] ( I ( Ax h ) - I0 ( x ) E A, h ) = + 2 x e R Affine: SC G SI CET BAHH ST S BJ HB BL LK

More Related