3D Shape and Motion Recovery from Multiple Views

1. Previously • Two view geometry: epipolar geometry • Stereo vision: 3D reconstruction epipolar plane epipolar lines epipolar lines O’ O Baseline

2. Today Orthographic projection • Two views • 3 or more views: factorization – simultaneous recovery of motion and shape

3. Orthographic Projection (Reminder) • Parallel projection rays, orthogonal to image plane • Focal center at infinity

4. Two Views Implies that Eliminating Z Since Therefore

5. Epipolar lines This is a linear equation

6. Further Simplification • Select one point in first image and its corresponding point in the second image to be the origin of the two images • In this coordinate frame translation is 0 • Expression for epipolar lines:

7. Epipolar Line Recovery • We need 4 corresponding points: • 1 to eliminate translation • 3 to determine the 4 components of R up to scale • The rest of the components cannot be determined • In particular, cannot be determined from , because these components are known only up to scale

8. Shape Recovery from Two Views • Perspective: • Translation recovered up to scale • 3D shape recovered up to scale • Recovery only if non-zero translation • No calibration – recovery up to a projective transformation (“projective shape”) • Orthographic: • Rotation along epipolar line cannot be recovered • 3D shape cannot be recovered • Recovery is possible up to an affine transformation (“affine shape”) • Recovery only if non-trivial rotation • Translation along line at infinity = rotation

9. Recovery from Three Views • Under orthographic projection metric recovery is possible from three views • Only rotation matters • Rotation has three degrees of freedom • Given an image, one rotation is in the image and two are out of plane rotations • Ignoring the in-plane rotation we can associate the image with a point on the unit sphere

10. Recovery from Three Views Im3 Im1 Im2

11. Im3 b g a a Im1 c b Im2 Recovery from Three Views

12. b g a a b c Recovery from Three Views • a, b, c are unknown • a, b, g are known – angles between epipolar lines • Can we determine lengths from angles?

13. g b a Recovery from Three Views • In the plane the angles determine the sides of a triangle up to scale g b a

14. b g a a b c Recovery from Three Views • On a sphere the sides are determined completely by the angles • Therefore three views determine all the components of rotation (up to reflection) • Once the rotation is known structure can be recovered

15. Factorization • Simultaneous recovery of shape and motion • Input: • A video sequence • Feature points tracked • Assumptions: • Rigid scene • Orthographic projection • All tracked points appear in all frames • Observation: tracked point locations satisfy linear relations that can be exploited for robust recovery

16. Singular Value Decomposition (SVD) • Every real matrix can be decomposed to a product of three matrices: • With • D diagonal • U,V orthonormal U orthonormal basis to row space of MV orthonormal basis to column space of M

17. Singular Value Decomposition (SVD) or Are called singular values

18. Relation to Eigenvalue Problems

19. Singular Value Decomposition (SVD) Rank k least squares approximation of M • Example: k=3 • Take the 3 largest singular values: • Rank 3 approximation of M:

20. Factorization • Goal: given p corresponding points in f frames, compute the 3-D location of each point and the transformation between the frames Measurements Transformation Shape (3-D locations)

21. Factorization • Step 1: eliminate translation Set the centroid of the points in each frame to be the origin Now

22. Factorization Constructing M :

23. Factorization Constructing M :

24. Factorization Constructing M :

25. Factorization Constructing M :

26. Factorization Constructing M :

27. Factorization Goal: given M, find S and T What should rank(M) be?

28. Factorization Goal: given M, find S and T • Compute the SVD of M • rank(M) should be 3, since rank(T)=rank(S)=3 • Noise cleaning: find the rank 3 approximation of M using the 3 largest singular values

29. Factorization • So far: • Define • The decomposition can now be written as • Factorization is not unique, since , A invertible

30. Factorization • T should contain valid rotations • 3f equations, 6 unknowns: • Each row is defined as • And should maintain

31. Factorization • is 3x3, symmetric • Linear system of equations in 6 unknowns • Once B is recovered it can be factored to find A • Solution is unique up to a global rotation

32. Factorization • Eliminate translation, produce M • Use SVD to find the rank 3 approximation of M • Solution ambiguous, up to an invertible matrix • Find matrix A such that T contains valid rotations • Solution is unique (up to a global rotation)

33. Factorization • Advantages • Simultaneous recovery of shape and motion • Simple algorithm, based on linear equations • Robust to noise • Disadvantages • Orthographic projection • All points should appear in all frames (factorization with missing data is difficult)

35. Summary • Shape and motion recovery under orthographic projection • Two views: • Parallel epipolar lines • 4 corresponding points are needed • Recovery of affine shape • Three or more views • Metric recovery • Simultaneous recovery of shape and motion using SVD factorization