Axial Light Field for Curved Mirrors: Reflect Your Perspective, Widen Your View

1. Axial Light Field for Curved Mirrors:Reflect Your Perspective, Widen Your View Yuichi Taguchi Amit Agrawal Srikumar Ramalingam Ashok Veeraraghavan Mitsubishi Electric Research Labs (MERL)

2. Non-Single Viewpoint Image(Spherical Mirror + Perspective Camera)

3. Single-Viewpoint Image(Cube Map, FOV 140)

4. No Approximation, No Knowledge of Scene Geometry

5. Single/Non-Single Viewpoint Perspective Camera Perspective Camera Hyperbolic Mirror Virtual Viewpoint Spherical Mirror Locus of Viewpoint (Caustic) Perspective Camera Catadioptric System (Mirror + Camera) Single Viewpoint Single Viewpoint Non-Single Viewpoint

6. Perspective Camera Orthographic Camera Foci Virtual Viewpoint Hyperbola/Ellipse Parabola Single-Viewpoint Catadioptric Systems [Baker & Nayar 99] • Only a few single-viewpoint configurations • Other common configurations lead to non-single viewpoint • Spherical mirror • Parabolic mirror with perspective camera • Hyperbolic/elliptic mirror with perspective camera not on foci

7. Scene Prior (e.g. Plane) Generating Single-Viewpoint Image fromNon-Single Viewpoint Image • Distortion Correction Approaches • Use one image • Single-viewpoint approximation • Use scene prior [Swaminathan et al. 03] • Generating exact perspective views is impossible without knowing scene geometry • Our Approach • Use multiple images • Capture all the rays required to generate an exact single-viewpoint image • Does not require scene prior and 3D reconstruction

8. Light Field [Levoy & Hanrahan 96, Gortler et al. 96] • If we capture all the rays that pass through the virtual viewpoint, we can generate a single-viewpoint image • What is the best possible sampling? Light Field Plane Mirror Surface Virtual Viewpoint

9. Input Images (Axial Light Field)

10. Input Output Copy Circles

11. Single-Viewpoint Image(Cube Map, FOV 140)

12. Key Idea • Rotationally symmetric mirrors • Capture axial light field • Move the camera along the mirror axis • Exact single-viewpoint image generationwithout scene prior and 3D reconstruction • Better sampling than typical planar light field Symmetry Axis AxialLight Field Planar Light Field

13.  d  Geometric Interpretation A cone of rays in virtual camera(Angle ) A cone of raysin real camera(Distance d, Angle ) Rotationally Symmetric Mirror Virtual Viewpoint Axial camera [Ramalingam et al. 06](All the rays pass through the axis)

14. v u x Light Fields for Symmetric Mirrors in 3D x-u-v Slice y x v u

15. v u x Light Fields for Symmetric Mirrors in 3D y=0 x-u-v Slice y x Virtual Viewpoint Planar Light Field Sampling Spherical Mirror

16. Light Fields for Symmetric Mirrors in 3D y0 x-u-v Slice y x v Virtual Viewpoint u x Planar Light Field Sampling Spherical Mirror

17. Light Fields for Symmetric Mirrors in 3D (x,y)=(uz,vz) x-u-v Slice y x v Virtual Viewpoint u x Axial Light Field Sampling Spherical Mirror

18. Light Fields for Symmetric Mirrors in 3D (x,y)=(uz,vz) u-v Slice y Image Plane x v Virtual Viewpoint u Axial Light Field Sampling Spherical Mirror

19. Planar LF vs. Axial LF • Planar Light Field • Sample a 4D subset of rays • The camera captures a cone of rays only when placed on the mirror axis • Other cameras capture only a few rays • Axial Light Field • Sample a 3D subset of rays • Every camera captures a cone of rays • Light rays required to generate a single-viewpoint image are concentrated in this 3D subset

20.  d  Axial LF Sampling Parameters Real Camera A cone of rays in virtual camera(Angle ) A cone of raysin real camera(Distance d, Angle ) Virtual Camera

21. Location of real camera (d) Ray angle in real camera () [] Resolution Reduction Real Camera FOV Ray angle in virtual camera () [] Ray angle in virtual camera () [] Mirror Shapes Sphere Parabola Concave Sphere Virtual camera Cone Axial LF Sampling Parameters

22. Single-Viewpoint Image Generation Copy Resized Circles A cone of rays in virtual camera(Angle ) A cone of raysin real camera(Angle , Distance d) Virtual Perspective Image Input Images

23. Limited Input FOV InvalidZ position Simulation Results for Different Mirror Shapes Concave Sphere Sphere Parabola Cone Input Far Input Near FOV 24 Output FOV 140

24. Circle radius in real image [pixels] Mag. factor of focal length 300 5 250 4 200 3 150 100 2 50 1 0 0 50 100 150 200 250 300 0 50 100 150 Circle radius in virtual image [pixels] Location of real camera Changing Resolution Property • Change focal length (zoom) for each capture position • Change resolution property without changing mirror shapes To achieve resolution propertysimilar to a perspective camera Spherical Mirror Constant focal length Variable focal length

25. Setup Robot Arm Mirror Ball Comparison Planar Light Field Axial Light Field Same Number (25) of Input Images (FOV 32 x 24)

26. Comparison Planar Light Field Axial Light Field Aliasing /Ghosting Output Cube Maps (FOV 140)

27. Advantages • Axial light field based catadioptric imaging • Exact single-viewpoint image generation in wide angle • Without any scene prior, 3D reconstruction • For any rotationally symmetric mirror • Offer changing resolution property • Without changing mirror shape • Virtual viewpoints can be varied on the mirror axis

28. Limitations • Static scene • Precise camera motion along the mirror axis • Implementation using multiple cameras would be difficult

29. Summary • Analysis of light rays for rotationally symmetric mirrors • Axial light field sampling • Exact single-viewpoint image generation