Theory and Practice of Projective Rectification Richard I. Hartley

1. Theory and Practice of Projective RectificationRichard I. Hartley Presented by Yinghua Hu

2. Goal • Apply 2D projective transformations on a pair of stereo images so that the epipolar lines in resulting images match and are parallel to the x-axis.

3. Epipolar Line u’ Y2 X2 Z2 O2 Epipole Stereo ConstraintsEpipolar Geometry M Image plane Y1 u O1 Z1 X1 Focal plane

4. Before rectification

5. Epipolar lines

6. After rectification

7. Some terms • Cofactor matrix A* • A*A = AA*=det(A)I • If A is an invertible matrix, A*≈(AT)-1 • Skew symmetrix matrix • Given a vector t = (tx, ty, tz)T [t]× = [ 0 -tz ty tz 0 -tx -ty tx 0 ] • [t]×s = t×s

8. Fundamental matrix x’TFx = 0

9. Some Properties • F is the fundamental matrix corresponding to an ordered pair of images (J, J’) and p and p’ are the epipoles, then • FT is the fundamental matrix corresponding to images (J’, J) • F=[p’]×M=M* [p]×, M is non-singular and not unique • Fp=0 and p’TF=0

10. Outline of the algorithm • Identify image-to-image matches ui↔ ui’ between the two images. • Compute the fundamental matrix F and find epipoles p and p’ by Fp=0 and p’TF=0. • Select a projective transformation H’ that maps the epipole p’ to the point at infinity. • Find the matching projective transformation H that minimizes the disparity of the corresponding points. • Resample the two images J and J’ and generate the rectified images

11. Mapping epipole p’ to infinity • Required mapping H’ = GRT • T translate a selected point u0 to the origin • R rotate about the origin taking p’ to (f,0,1)T • G maps rotated p’ to infinity (f,0,0) • The composite mapping H’ is to the first order a rigid transformation in the neighborhood of u0 • Choice of u0, center of the image or the center of all the corresponding points

12. Matching Transformations • The matching transformation H on the first image is of the form • H = (I+H’p’aT)H’M, in which M can be computed by solving F=[p’]×M and a is a vector. • It is shown that I+H’p’aT is an affinetransformation of the form [x y z 0 1 0 0 0 1] • Least square minimization is used to find x,y,z which will minimize ∑d(Hui, H’ui’)2, so the disparity in x direction is also minimized.

13. Non Quasi-affine Transformation

14. Quasi-affine Transformation • The line at infinity L∞ in the projective plane P2 consists of all points with last coordinate equal to 0. • A view window is a convex subset of the image plane. • A projective transformation H is quasi-affine with respect to view window W if H(W)∩L∞ = Ø • Theorem 5.7 proves that for a quasi-affine transformation H’ on W’ mapping p’ to infinity, and H be any matching projective transformation of H’ on W, there exists a subset W+ of view window W, such that H is quasi-affine with respect to W+ and W+ contains all the matching points in W corresponding to W’. W- W+

15. Resampling images • Firstly, determine the dimensions of the output images, a rectangle R containing H’(W’)∩H(W+). (When I implement it, I use a region containing H’(W’) U H(W+)). • For each pixel location in R, applying separately inverse transformation H-1 and H’-1 to find the corresponding location in images J and J’, interpolate the color values to get the rectified image pair. • Linear interpolation is adequate in most cases.

16. Scene Reconstruction • Without camera parameters, the scene can be reconstructed but there will remain projective distortion. • It is possible that the reconstructed images will be disconnected. • To avoid this problem, a translation is done in x direction in one of the images before reconstruction.

17. Applications • Rectification may be used to detect changes in the two images, simplifying image matching problem.

18. Main contribution • Rectify images based on corresponding points alone • Give a firm mathematical basis as well as a rapid practical algorithm

19. Doubt in this paper • Page 8, below equation 7 Hp = (I+H’p’aT)H’Mp = (I+H’p’aT)H’p’ How? → = (I+aTH’p’)H’p’ ≈ H’p’ H’- 3×3 p’- 3×1 a - 3×1 H’p’aT - 3×3 aT- 1×3 aTH’p’ - 1×1