1 / 51

Structure from motion

Structure from motion. Marc Pollefeys COMP 256. Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …. Last time: Optical Flow. I x u. I x. u. I x u= - I t. I t. Aperture problem:. two solutions: - regularize (smoothness prior) constant over window

alohilani-matthew
Download Presentation

Structure from motion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …

  2. Last time: Optical Flow Ixu Ix u Ixu=- It It Aperture problem: • two solutions: • - regularize (smoothness prior) • constant over window • (i.e. Lucas-Kanade) Coarse-to-fine, parametric models, etc…

  3. Tentative class schedule

  4. Today’s menu • Affine structure from motion • Geometric construction • Factorization • Projective structure from motion • Factorization • Sequential

  5. Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990).  1990 Optical Society of America. • Given m pictures of n points, can we recover • the three-dimensional configuration of these points? • the camera configurations? (structure) (motion)

  6. Orthographic Projection Parallel Projection

  7. Weak-Perspective Projection Paraperspective Projection

  8. Problem: estimate the m 2x4 matrices M and the n positions P from the mn correspondences p . i j ij The Affine Structure-from-Motion Problem Given m images of n fixed points P we can write j 2mn equations in 8m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

  9. If M and P are solutions, i j The Affine Ambiguity of Affine SFM When the intrinsic and extrinsic parameters are unknown So are M’ and P’ where i j and Q is anaffine transformation.

  10. Affine Spaces: (Semi-Formal) Definition

  11. 2 Example: R as an Affine Space

  12. In General The notation is justified by the fact that choosing some origin O in X allows us to identify the point P with the vector OP. Warning:P+u and Q-P are defined independently of O!!

  13. Barycentric Combinations • Can we add points? R=P+Q NO! • But, when we can define • Note:

  14. Affine Subspaces

  15. Affine Coordinates • Coordinate system for U: • Coordinate system for Y=O+U: • Affine coordinates: • Coordinate system for Y: • Barycentric • coordinates:

  16. When do m+1 points define a p-dimensional subspace Y of an n-dimensional affine space X equipped with some coordinate frame basis? Rank ( D ) = p+1, where Writing that all minors of size (p+2)x(p+2) of D are equal tozero givestheequationsof Y.

  17. Affine Transformations • Bijections from X to Y that: • map m-dimensional subspaces of X onto m-dimensional • subspaces of Y; • map parallel subspaces onto parallel subspaces; and • preserve affine (or barycentric) coordinates. • Bijections from X to Y that: • map lines of X onto lines of Y; and • preserve the ratios of signed lengths of • line segments. 3 In E they are combinations of rigid transformations, non-uniform scalings and shears.

  18. Affine Transformations II • Given two affine spaces X and Y of dimension m, and two • coordinate frames (A) and (B) for these spaces, there exists • a unique affine transformation mapping (A) onto (B). • Given an affine transformation from X to Y, one can always write: • When coordinate frames have been chosen for X and Y, • this translates into:

  19. Affine projections induce affine transformations from planes onto their images.

  20. Affine Shape Two point sets S and S’ in some affine space X are affinely equivalentwhen there exists an affine transformation y: X X such that X’ = y ( X ). Affine structure from motion = affine shape recovery. = recovery of the corresponding motion equivalence classes.

  21. Geometric affine scene reconstruction from two images (Koenderink and Van Doorn, 1991).

  22. Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990).  1990 Optical Society of America. (Koenderink and Van Doorn, 1991)

  23. The Affine Epipolar Constraint Note: the epipolar lines are parallel.

  24. Affine Epipolar Geometry

  25. The Affine Fundamental Matrix where

  26. An Affine Trick.. Algebraic Scene Reconstruction

  27. The Affine Structure of Affine Images Suppose we observe a scene with m fixed cameras.. The set of all images of a fixed scene is a 3D affine space!

  28. has rank 4!

  29. From Affine to Vectorial Structure Idea: pick one of the points (or their center of mass) as the origin.

  30. What if we could factorize D? (Tomasi and Kanade, 1992) Affine SFM is solved! Singular Value Decomposition We can take

  31. From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ? Note:Absolute scale cannot be recovered. TheEuclideanshape (defined up to an arbitrary similitude) is recovered.

  32. Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).  1991 IEEE.

  33. More examples Tomasi Kanade’92, Poelman & Kanade’94

  34. More examples Tomasi Kanade’92, Poelman & Kanade’94

  35. More examples Tomasi Kanade’92, Poelman & Kanade’94

  36. Further Factorization work Factorization with uncertainty Factorization for indep. moving objects Factorization for dynamic objects Perspective factorization (next week) Factorization with outliers and missing pts. (Irani & Anandan, IJCV’02) (Costeira and Kanade ‘94) (Bregler et al. 2000, Brand 2001) (Sturm & Triggs 1996, …) (Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective))

  37. Multiple indep. moving objects

  38. Multiple indep. moving objects

  39. Dynamic structure from motion (Bregler et al ’00; Brand ‘01) Extend factorization approaches to deal with dynamic shapes

  40. Representing dynamic shapes (fig. M.Brand) represent dynamic shape as varying linear combination of basis shapes

  41. Projecting dynamic shapes (figs. M.Brand) Rewrite:

  42. Dynamic image sequences One image: (figs. M.Brand) Multiple images

  43. Dynamic SfM factorization? Problem: find J so that M has proper structure

  44. Dynamic SfM factorization (Bregler et al ’00) Assumption: SVD preserves order and orientation of basis shape components

  45. Results (Bregler et al ’00)

  46. Dynamic SfM factorization (Brand ’01) constraints to be satisfied for M constraints to be satisfied for M, use to compute J hard! (different methods are possible, not so simple and also not optimal)

  47. Non-rigid 3D subspace flow (Brand ’01) • Same is also possible using optical flow in stead of features, also takes uncertainty into account

  48. Results (Brand ’01)

  49. Results (Brand ’01)

  50. Results (Bregler et al ’01)

More Related