Image-Based Rendering from a Single Image

1. Image-Based Rendering from a Single Image Samuel Boivin – Andre Gagalowicz INRIA

2. Introduction To recover an approximation of BRDF of surface from a single image. (including specular, isotropic or anisotropic surfaces) Hierarchical , interactive technique using error between the rendered image and the real image.

3. Background Camera parameter (3D geometrical model, 2D image) [DeMenthon and Davis] Model-based object pose in 25 line of code – ECCV 92 BRDF Model: [Ward] Measuring and modeling anisotropic reflection – SIGGRAPH 92 Radiance map: [Tumblin and Rushmeier] Tone reproduction for realistic image – IEEE Computer Graphics and Applications November 1993. How to find light pose from a single image

4. Note: Images record color, Radiance maps record brightness

5. Background and Related Work 1. Reflectance Recovery using a Specific Device. - Estimate the five parameters of anisotropic BRDF model. [Ward] Measuring and modeling anisotropic reflectance – SIGGRAPH 92 2. Reflectance Recovery from Several Images. - Method without Global Illumination. - Method with Global Illumination. 3. Reflectance Recovery from a Single Image. - Method without Global Illumination. - Method with Global Illumination. - Radiosity-based Algorithm

6. Elements of Reflectance Recovery • Notion of Group • - Input : 3D geometrical model, a single image • - Extraction of the object reflectance from the pixel. • (by the projection of these objects in the image) • - Problems (using a single image) • -A lot of surfaces are not visible. • - Notion of Group • - The object and the surface have a same reflectance property • - Manual operation (Geometrical modeling process)

7. Elements of Reflectance Recovery • Reflectance Model and Data Description • Image-Based Modeling (Alias | Wavefront’s Maya modeler) • Camera parameters • [DeMenthon and Davis] Model-based object pose in 25 line of code – ECCV 92 • Photometric recovery method • [Ward] Measuring and modeling anisotropic reflection – SIGGRAPH 92 • Five parameters for a complex BRDF: • Diffuse ρd, Specular ρd, anisoptropic direction (x), anisotropic roughness (ax, ay)

8. Elements of Reflectance Recovery 3D Geometrical Modeling • Geometrical model • - object, camera, • light sources poses • Phothmetric model • - reflectance, • light source intensity • Synthetic image using a classical rendering Software (Phoenix – global illumination software)

9. Overview of the Algorithm

10. Inverse Rendering from a Single Image • Case of perfect diffuse surface • Diffuse reflectance : the average of radiances covered by the projection of the group in the original image. • Textured surface (using a pure diffuse) : Create a good visual approximation.

11. Inverse Rendering from a Single Image Case of perfect diffuse surface Error between the original and the rendered image Where, B : average radiance P : pixels covered by the projection of object j in the original image. T( ) is camera transfer function ( - correction function) • Camera transfer function : To convert light input into electrical • (analog or digital) signals.

12. Inverse Rendering from a Single Image Diffuse reflectance ρd of object j is proportional to the average radiance B The function f () eliminates problems by smaller object. Textures are not take into account – only consider a diffuse reflectance parameter ρd . The radiances, the emittances and the full geometry (form factors)  Solve radiosity equation for the reflectance.

13. Inverse Rendering from a Single Image Case of perfect and non-perfect specular surface Diffuse hypothesis failed  considered as a perfect mirror. Perfect specular surface -The easiest case to solve (ρd =0, ρs = 1), Need not iteration Non-perfect specular surface - Require iteration to obtain an optimum ρs

14. Inverse Rendering from a Single Image Case of both diffuse and specular surfaces with no roughness

15. Inverse Rendering from a Single Image • Case of isotropic surfaces • Recover Diffuse (ρd), Specular (ρd)and roughness (a) using Ward’s BRDF model. • Case of anisotropic surfaces • Most complicate case • Anisotropic model of Ward requires to minimize a function of 5 parameters. • (Diffuse ρd, Specular ρd, anisoptropic direction (x), anisotropic roughness (ax, ay)

16. Inverse Rendering from a Single Image • Case of textured surfaces • - Extracting the texture from image is an easy task • [Wolberg] Digital Image Warping – IEEE Computer Society Press • - Consider that it already has received the energy from the light source. •  Otherwise, over-illuminated. • - Radiosity texture : balances the extracted texture with an intermediate texture • in order to minimize the error (the real and synthetic image) • - Case of perfect diffuse surface • Texture is computed by an iterative method. • - At the first, extract from the real image. • - Synthetic image • - Multiplied by the ratio (newly texture of synthetic / texture of real image)

17. Inverse Rendering from a Single Image • Case of textured surfaces • The problems • - A texture including the shadows, the specular reflection and the highlight. • - It is extremely hard to solve using a single image.

18. Results

19. Results

20. Future Works • [Debevec] Efficient View-Dependent Image-Based Rendering with Projective • Texture-Mapping – EGRW 98 • [Debevec] Rendering Synthetic Objects into Real Scenes - SIGGRAPH 98 • [Debevec] Modeling and Rendering Architecture from Photographs –SIGGRAPH96 • [Ward] Measuring and modeling anisotropic reflection – SIGGRAPH 92 • Fixed Camera – Obtain multiple images (Different exposure, light poses) • Assume perfect diffuse surface – extract texture (iterative method) –Render image. • Find radiance map – Estimate BRDF – Render image.