1 / 34

SIFT Flow: Dense Correspondence across Scenes and Its Applications

SIFT Flow: Dense Correspondence across Scenes and Its Applications. Ce Liu, Jenny Yuen, and Antonio Torralba, MIT PAMI 2011 (ECCV 2008) Presented by Han Hu, I-vision Lab. Ce Liu, Researcher at Microsoft Research New England Experience:

mayes
Download Presentation

SIFT Flow: Dense Correspondence across Scenes and Its Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SIFT Flow: Dense Correspondence across Scenes and Its Applications Ce Liu, Jenny Yuen, and Antonio Torralba, MIT PAMI 2011 (ECCV 2008) Presented by Han Hu, I-vision Lab

  2. Ce Liu, Researcher at Microsoft Research New England Experience: 1995-2002 Bachelor and Master student in Automation, Tsinghua University. 2002-2003 Assistant Researcher, Microsoft Research Asia. 2003-2009 PhD student in MIT. Main Works: Face Analysis

  3. Ce Liu, Researcher at Microsoft Research New England Experience: 1995-2002 Bachelor and Master student in Automation, Tsinghua University. 2002-2003 Assistant Researcher, Microsoft Research Asia. 2003-2009 PhD student in MIT. Main Works: Denoising and Super resolution

  4. Ce Liu, Researcher at Microsoft Research New England Experience: 1995-2002 Bachelor and Master student in Automation, Tsinghua University. 2002-2003 Assistant Researcher, Microsoft Research Asia. 2003-2009 PhD student in MIT. Main Works: Recognition and Parsing

  5. Ce Liu, Researcher at Microsoft Research New England Experience: 1995-2002 Bachelor and Master student in Automation, Tsinghua University. 2002-2003 Assistant Researcher, Microsoft Research Asia. 2003-2009 PhD student in MIT. Main Works: Motion Analysis

  6. Image Correspondence(Parametric Motion)

  7. Image Correspondence (1D Stereo Disparity)

  8. Image Correspondence (2D Optical Flow)

  9. Image Correspondence (Object Level Matching) [Berg et al. CVPR’05]

  10. Scene Level Correspondence

  11. Scene Level Correspondence

  12. Scene Level Correspondence

  13. Scene Level Correspondence

  14. How to establish dense scene correspondence? Input Support Before alignment

  15. How to establish dense scene correspondence? Input Support Optical flow Flow visualization code

  16. How to establish dense scene correspondence? Input Support Warping of optical flow

  17. How to establish dense scene correspondence? Input Support SIFT flow Dense SIFT image (RGB = first 3 components of 128D SIFT)

  18. How to establish dense scene correspondence? Warping of SIFT flow Dense SIFT image (RGB = first 3 components of 128D SIFT)

  19. Dense SIFT Flow Dense SIFT image (RGB = first 3 components of 128D SIFT

  20. Formulation • Energy function: Data term Small displacement bias Smoothness term p, q: grid coordinate, w: flow vector, u, v: x- and y-components, s1, s2: SIFT descriptors

  21. How to solve it?(Dual-Layer Belief Propagation) Complexity: O(n2h4) O(n2h2)

  22. Other Tricks to Speed Up(Coarse-to-Fine) Complexity: O(n2h2) O(n2logh)

  23. Coarse-to-Fine vs. 1-level Matching • Computational Time • 31s (Coarse-to-Fine) vs. more than 2 hours (1-level) for 256x256 images • Energy Cost

  24. How to Find Meaningful Image Pairs? Optical Flow: Dense Sampling in Time Sift Flow: Dense Sampling in the Space of All Images

  25. Experiments and Applications • Video Retrieval • Motion Prediction from a Single Image • Motion Synthesis • Image Registration • Face Recognition • Scene Parsing • …

  26. Video Retrieval

  27. Dense Scene Alignment

  28. Motion Prediction from a Single Image Warped Motion Original Image Best Match and Temporal Motion Ground Truth

  29. Motion Prediction Original Image Synthesized Sequence The top video match

  30. Image Registration

  31. Face Recognition ORL Database Compare with the-state-of-the-art

  32. Scene Parsing RGB SIFT Query tree sky Annotation SIFT RGB SIFT flow road field Nearest neighbors car unlabeled [C. Liu et al, CVPR’09]

  33. Scene Parsing Parsing Ground truth RGB SIFT Query Good matches  higher weight; CRF tree sky Annotation SIFT RGB SIFT flow road field Warpednearest neighbors car unlabeled

  34. Summary • Introduce a concept: Dense Scene Alignment • A basic tool for applications which need correspondences between images

More Related