Image Based Modeling and Rendering (PI: Malik)

1. Image Based Modeling and Rendering (PI: Malik) • Capture photorealistic models of real world scenes that can support interaction • Vary viewpoint • Vary lighting • Vary scene configuration • This objective was achieved by combining and inventing new techniques in computer vision and graphics.

2. Image-based Modeling and Rendering • 1st Generation---- vary viewpoint but not lighting • Acquire photographs • Recover geometry (explicit or implicit) • Texture map

3. Image-based Modeling and Rendering • Photographs are not Reflectance Maps ! • 2nd Generation---- vary viewpoint and lighting for non-diffuse scenes • Recover geometry • Recover reflectance properties • Render using light transport simulation Illumination Radiance Reflectance

4. Image-based Modeling • 2nd Generation---- vary viewpoint and lighting • Recover geometry & reflectance properties • Render using light transport simulation or local shading Original Lighting & Viewpoint Novel Lighting & Viewpoint

5. Inverse Global Illumination (Yu et al, SIGGRAPH 99) Reflectance Properties Radiance Maps Light Sources Geometry

6. Real vs. Synthetic

7. Real vs. Synthetic

8. Key contributions in Inverse Global Illumination • A digital camera can undertake all the data acquisition tasks involved. • Both specular and high resolution diffuse reflectance properties can be recovered from photographs. • Reflectance recovery can re-render non-diffuse real scenes under novel illumination as well as from novel viewpoints.

9. Image-based Modeling and Rendering • 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting Novel Viewpoint Novel Viewpoint & Configuration

10. Input Multiple range scans of a scene Multiple photographs of the same scene Output Geometric meshes of each object in the scene Registered texture maps for objects Framework

11. Segmentation Results[Yu, Ferencz and Malik TVCG ’01]

12. Models of Individual Objects

13. Texture-Mapping and Object Manipulation

14. Transitions • Façade has inspired several image-based modeling packages including Canoma from Metacreations. • Technique for registering range and radiance images being used by Cyra, a leading time of flight laser scanner company.

15. Continuing Challenges • Finding correspondences between widely separated camera views automatically • Models of reflectance and texture for natural materials and objects