Line segment tracking for 3D reconstruction

1. Line segment tracking for 3D reconstruction

2. Contents 1. Introduction 2. Feature Tracking 1. Points 2. Segments 3. 3D Reconstruction 1. Projective Reconstruction 2. Quasi-Affine reconstruction 3. Euclidean reconstruction

3. Introduction • Tracking features in an image sequence • Point : Lucas-Kanade method • Line : using novel approach • 3D reconstruction • Meshing, Dense matching • Line segment tracking for 3D reconstruction • Report submitted for the third year of Diplôme d’Ingénieur en Informatique et Communication , Laurent Guichard, August 2001

4. The general scheme for the reconstruction process

5. Feature Tracking Brightness Constancy

6. Brightness Constancy • Derive from Talyer series expansion • 한 점에서의 함수 값을 다른 한 점에서의 함수 값과 그 도함수로 예측하는 방법

7. Brightness Constancy Derive from Temporal derivates (Chain rule) Two unknowns brightness constancy constraint equation

8. Least-Square Fit • Brightness Constancy Equation : • Minimize the function : • Least square solution N2 x2 matrix

9. Results: Lucas-Kanade Image 0 Image 1 The blue segments represent the displacement vectors.

10. Points weeding • The fundamental constraints ,

11. Points weeding • RANSAC (RANdom SAmple Consensus) The solid points are inliers, the open points are outliers (a) A least-squares fit to the point data (b) The dotted lines indicate the threshold distance

12. Result : Points weeding

13. Edges detection • Canny Edge detector • Convolute with a Gaussian filter to smooth • Convolute with the 2D first-derivative approximation • Run a non-maximum suppression • hysteresis thresholding

14. Segment fitting • Consider an edge :

15. Segment matching • Weeding : the geometric distance criteria where The best match is the one that minimizes the orthogonal distance and maximizes the overlap distance.

16. Segment matching • Weeding : The correlation criteria – we need to define the same area around the segments s and s'

17. Segment matching • Simple correlation • Change the reference frame Image reference  Window reference we need to make a bi- linear interpolation of the neighbor pixels

18. Result : Segment matching Results for the correlation criterion on the same images:

19. Projective Reconstruction The trifocal tensor :

20. Trifocal Tensor

21. Tensor notation (V–1) Points correspondence: (V–2)

23. Computation of the Trifocal Tensor • Create a 27×1 vector t containing the entries of the trifocal tensor T . • From a set of points matches and lines matches fill the n×27 matrix A with equations (V–1) and (V–2). • Solve the system At=0by minimizing with • compute the SVD of A, A=UDVT , and choosing t equal to the last row of V

24. The camera matrices Extract the camera matrices.

25. Reconstruct 3D points

26. Reconstruct 3D points

27. Reconstruct 3D points –

28. Reconstruct 3D segments

29. Reconstruct 3D segments As one can see, some segments are incorrect. The bad segments are all close to the epipolar lines. Such a reconstruction is almost singular and gives large errors. The only way to solve this problem is to remove the segments that are closed to an epipolar line.

30. Reconstruct 3D segments The endpoints of the segments do not always match perfectly as show in the pictures below, extracted from the images 0 and 1. To solve this lack of precision, for each matched segment, we compute the correlation between their endpoints such that they will perfectly match.

31. The reconstruction ambiguity In the view 0, we can identify the scene but in the others views, it is difficult recognize it.

32. The reconstruction ambiguity (a) An image of a comb, (b) the result of applying a projective transform to the image The projective transformation does not preserve the convex hull of the set of points

33. Quasi-affine reconstruction • Quasi-affine • Let B be a subset of Rn and let h be a projectivity Pn, the projectivity h is said to be “quasi-affine” with respect to the set B if h preserves the convex hull of the set B • Cheirality • the point X lies in front of the camera P if and only if .

34. Quasi-affine reconstruction

35. Quasi-affine reconstruction Those images are obtained by projected the 3D points and 3D segments in the image using the cameras matrices.

36. Euclidean Reconstruction project to the dual image of the absolute conic Specifying linear constraints on – From five images a total of 10 equations are obtained

37. Questions