1 / 20

Image-Based Modeling and Rendering Overview

This overview provides an introduction to image-based modeling and rendering, including the concept of the plenoptic function and its limitations. It also explores various methods for capturing and rendering scenes using images, such as texture maps, image-depth pairs, and light probes. Additionally, it discusses the challenges and techniques involved in estimating depth and silhouette information from images. The use of light stages and light control methods is also examined.

chamness
Download Presentation

Image-Based Modeling and Rendering Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 395: Adv. Computer Graphics Overview: Image-Based Modeling and Rendering Jack Tumblin jet@cs.northwestern.edu

  2. GOAL: First-Class Primitive • Want images as ‘first-class’ primitives • Useful as BOTH input and output • Convert to/from traditional scene descriptions • Want to mix real & synthetic scenes freely • Want to extend photography • Easily capture scene:shape, movement, surface/BRDF, lighting … • Modify & Render the captured scene data • --BUT-- • images hold only PARTIAL scene information • You can’t always get what you want”–(Mick Jagger 1968)

  3. Plenoptic Function (Adelson, Bergen `91) • for a given scene, describe: • ALLrays through • ALLpixels, of • ALL cameras, at • ALL wavelengths, • ALL time F(x,y,z,,,, t) “Eyeballs Everywhere” function (5 or 7-D!) … … … … … … … … … … …

  4. ‘Scene’ causes Light Field Light field: holds all outgoing light rays Shape, Position, Movement, Emitted Light Reflected, Scattered, Light … BRDF, Texture, Scattering Scene modulates outgoing light; light field captures it all.

  5. A Big Light Field Question: Image entraps a partial scene description,… • Computer Vision problem: 3D->2D • Image point  scene surface point (usually) • Occlusion hides some scene surfaces • (BRDF * irradiance) tough to split apart! ? Does Plenoptic fcn. contain full scene ? • Exhaustive record of all image rays • Even SIMPLEST scene is huge, redundant, • The ‘consequences’ of all possible renderings* so

  6. A Big Light Field Answer: !NO! Image entraps a partial scene description • Many-to-One map; 3D->2D • Occlusion hides some scene features • (BRDF * irradiance) tough to split! • limited resolution ? Does Plenoptic fcn. contain full scene ? Two Options for light field methods: • Find a limited subset of scene info, • Use MORE than plenoptic function data: (vary lights, etc.)

  7. Shape Problems: Correspondence Can you find ray intersections? Or ray depth? Correspondence Problem: Ray colors might not match for non-diffuse materials (BRDF)

  8. Shape Problems: Correspondence Can you find ray intersections? Or ray depth? Correspondence Problem: Ray colors might not match for non-diffuse materials (BRDF)

  9. It gets worse… A ‘Circular problem’: PLUS! depth-of-focus, sampling, indirect illum… SurfaceNormal BRDF Shape Irradiance

  10. 8-to-10-Dimensional Ideal? Light field(4D) + light sources(4D) + time +  Emitted Light Shape, Position, Movement, Reflected, Scattered, Light … BRDF, Texture, Scattering

  11. Full Plenoptic Fcn? MAYBE... • Linking illumination rays to light field rays might do it; • Swaps ‘correspondence’ problem for ‘massive data problem’. • Shortcuts, simplifications? F(xc,yc,c,c,xl,yl l,l, , t) camera projector

  12. Practical IBMR What useful partial solutions are possible? • Texture Maps++: • Image(s)+Depth: (3D shell) • Estimating Depth & Silhouettes • ‘Light Probe’ measures real-world light • Light control measures BRDF • Hybrids: BTF, stitching, …

  13. Texture Maps ++ Re-use rendering results: ‘Impostors’, ‘Billboards’, ‘3D sprites’ • Render portion of scene as a texture • Apply to mesh or plane  to C.O.P.; • Replace if eyepoint changes too much

  14. Images + Depth • 1 Image + Depth: a ‘thin shell’ • Reprojection (well known); Z-buffers can help • McMillan`95: 4-way raster ensures depth order • Problem: ‘holes’, occlusion, matching • Multiple Images: • LDI, LDI trees for multiresolution • Limitations: • Presumes diffuse-only environment • Depth capture tough: laser TOF reflectometer, manual scanner, structured light, or …

  15. Estimating Depth, Silhouettes Mildly new IBMR methods can help… • Sparse, manual image correspondences (Debevec, Seitz,) • Video sequences with camera motion tracking • Image (silhouette)-based Visual Hulls, ‘voxel carving’ (VIDEO!) Mostly a Classic Computer Vision Problem: • Epipolar Geometry: reduce search for correspondences • Global & local tracking & alignment methods…

  16. Light Probe: Irradiance Estimate • Place mirrored ball in scene, • Photograph (careful! High contrast image!) • Unwrap: map image positionangle • Use as illumination source • Uses: • mixing real & synthetic objects (Ward 96) • separating reflectance & illum (Yu 97) • movies, movies, movies...

  17. Light Stage: Debevec2001 • Carefully control incoming light direction (light stages, whirling banks of lights, etc) • Images vs. light(,) • Weighted sum of imagesweighted sum of lights Debevec et al. 2001

  18. Light Stage Methods • Specular Component is Polarized • Scan surface geometry (before, during) • Scattered data interpolation to approx. BRDF. Debevec et al. 2001

  19. Light Control Methods • light probe gathers illum, • light stage gathers face response to light; • Real face artificially inserted into real scene

  20. Conclusion • Very active area • Heavy overlap with computer vision: careful not to re-invent & re-name! • Compute-intensive, but easily parallel; applies graphics hardware to broader probs.

More Related