What Does the Scene Look Like From a Scene Point?

1. What Does the Scene Look Like From a Scene Point? M. Irani, T. Hassner, and P. AnandanECCV 2002 Donald TanguayAugust 7, 2002

2. Overview • Categorization of novel view synthesis • Outline of approach • Planar parallax formulation • Synthesizing the virtual view • Practical simplification • Results • Assessment

3. Novel View Synthesis Breakdown into 3 categories: • 3D reconstruction • View transfer • Sampling methods

4. 3D Reconstruction • Fully reconstruct scene, then render view • Geometric error criteria do not translate well to errors in novel view • Reconstruction and rendering occur in different coordinate systems • Problems amplified with novel viewpoints significantly different from real cameras

5. “View Transfer” • For example: 2 images, dense correspondence, and trifocal tensor • Avoids reconstruction • Errors in correspondence • Synthesis uses forward warping step, which results in “hole-filling” at surface discontinuities • Problems amplified by severe changes in viewpoint

6. Sampling Methods • E.g., lightfield and lumigraph • Avoid reconstruction and correspondence • Require very large sampling of view-space • Data acquisition is problematic • Space-time costs are impractical

7. Features of Their Method • Avoids reconstruction, correspondence • Backward (“inverse”) warping avoids holes • Optimizes errors in coordinate system of novel view • Handles significantly different viewpoints • Small number of input images (~10)

8. Typical Scenario Choose a scene point Vfrom which to look.

9. Imaging Geometry Black point in eachimage is the virtualepipole (image ofselected COP).

10. Color Consistency Test If projections were aligned, matching determines the correct color: • However: • Only one correspondence is known – the virtual epipole • All other correspondences are warped because the lines are in different coordinate systems.

11. Overview of Approach • Choose virtual viewpoint V (a scene point) • For each pixel in the virtual image: • Calculate the line of sight L • Map images of L into a common coordinate system • Stack the colorings of L for comparison • Select the first consistent color as the color of the pixel

12. Mapping onto a Common Coordinate System One camera is selected as the reference camera R.Projections of L in all other cameras will be mapped into R’s image.

13. Imaging Camera 1 Transform the line in C1 into R by the homographyinduced by the ground plane.

14. Imaging Camera 3 Geometrically, the homography displaces each pixel in Ci asthough the corresponding 3D point was on the ground plane.The “piercing point” always maps to the same point in R.

15. Pencil of Lines inReference Camera After plane alignment, the lines in the reference camera fanfrom the imaged piercing point to the virtual epipoles.

16. Projective Geometry Review • Homography: • A.k.a. Collineation, projective transformation • In P3: 3x4 matrix with 11 degrees of freedom • Points and lines: • Point x lies on line lx†l  0 • Intersection of lines l and m is point plm • Line joining points p and q is lpq

17. Line Configuration In R’s image plane, whatis the relationship betweenblue and red lines?

18. Line Alignment red green Given real epipoles ei and virtual epipoles vi: for any axis point pV,Mi is the projective transformation that brings each line li into alignment with lR.

19. Virtual View Hsyn is the homography between the synthesized view and the reference view R. • Position is fixed by the virtual epipoles • Free parameters (can be specified in Hsyn): • Orientation (look direction) • Intrinsic parameters (e.g., zoom)

20. Virtual Epipoles In an uncalibrated setting, the position of the virtual camera can be specified in several ways: • Manually pin-point same scene point in all cameras. • Pin-point in two images and geometrically infer in others using “trifocal constraints.” • Pin-point in one image and use correlation techniques to find correspondence in others

21. Relating Virtual to Reference

22. Algorithm Outline For each pixel p in the synthesized image: • Find the imaged piercing point pv= Hsyn·p. • Align all imaged lines of sight using the line-to-line transformations Mi. • Find the first color-consistent column. • Assign pixel this color.

23. Color Consistency • Assume Lambertian objects. • A is a (n+1)3 matrix of the column of colors in YIQ color space. •  is the maximal eigenvalue of the covariance matrix of A. • Select first column with  under a threshold. • Paint with the median color of that column.

24. Important Details • Local “smoothing”: They prefer color consistent columns whose 3D position is spatially consistent with that of neighboring pixels. • Uniform regions: They flag used pixels in source images to prevent their repeated matching. • Pixel scanning order: They evaluate for physical points closer to ground plane first; then farther. • Ground subtraction: Except for piercing point, remove ground plane from coloring stack

25. Practical Simplification • Cameras are coplanar • Real epipoles lie on a line in R • Rectifying R into the “nadir view” makes the line of epipoles go to infinity • The Mi line-to-line transformations become affine – simple linear stretching of the lines

26. Synthetic Scene • Extreme change in viewpoint • Objects seen through gate, while source images have only floor seen through gate

27. Color Analysis

28. Folder Scene • Off-the-shelf digital camera, constant-height tripod • Triangle occludes distant folder • 11 images used for (e)

29. Puppet Scene • Green smear on lower left of (e): “This floor region was not visible in any of the input images.” • 9 images used for (e)

30. Assessment • Interesting use of projective, epipolar geometry • Needs only weak calibration • Needs failure analysis • How to define Hsyn? • Explicit notion of visibility could help • Manual selection among source images? • Observation: no occlusions in source imagery – hmm…