1 / 18

Image-Based Rendering

Image-Based Rendering. Geometry and light interaction may be difficult and expensive to model Think of how hard radiosity is Imagine the complexity of modeling the exact geometry of carpet (as just one example) Image based rendering seeks to replace geometry and surface properties with images.

cooper-jarvis
Download Presentation

Image-Based Rendering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image-Based Rendering • Geometry and light interaction may be difficult and expensive to model • Think of how hard radiosity is • Imagine the complexity of modeling the exact geometry of carpet (as just one example) • Image based rendering seeks to replace geometry and surface properties with images

  2. Texture Mapping • Replace small scale geometry and surface changes with an image attached to the surface • Define a mapping from surface points into a texture image • When the properties of that point are required, look up the pixel in the texture image • A research field in itself • Various methods of sampling the texture • Various ways of combining the samples with direct lighting calculations • Ways of arranging and accessing textures in memory

  3. Environment Mapping • Replace the world around an object with images • Generally used for fast reflection computations • Typically, define a cube around the object and associate images with the inside of each face • When seeking reflection color, shot ray from surface point onto cube and take corresponding image point • Images represent a view of the world from the center of the object • Assumes that the world wouldn’t look much different if viewed from points on the object’s surface

  4. Plenoptic Function • Describes the intensity of light: • passing through a given point, x • in a given direction, (,) • with given wavelength,  • at a given time, t • Many image-based rendering approaches can be cast as sampling from and reconstructing the plenoptic function • Note, function is generally constant along segments of a line (assuming vacuum)

  5. Image-Based Rendering • Aim: Generate an image from a desired view using existing images from other views • May or may not know the viewing parameters for the existing images • Existing images may be photographs or computer generated renderings

  6. A Plethora of Approaches • Methods differ in many ways: • The range of new viewpoints allowed • The density of input images • The representation for samples (known images) • The amount of user help required • The amount of additional information required (such as intrinsic camera parameters) • The method for gathering the input images

  7. Movie-Map Approaches • Film views from fixed locations, closely spaced, and store on video-disc (would be DVD now) • Allow the user to jump from location to location, and pan • Appropriate images are retrieved from disk and displayed • No reprojection – just uses nearest existing sample • Still used in video games today, but with computer generated movies

  8. Quicktime VR (Chen, 1995) • Movie-maps in software • Construct panoramic images by stitching together a series of photographs • Semi automatic process, based on correlation • Scale/shift images so that they look most alike • Works best with >50% overlap • Finite set of panoramas – user jumps from one to the other

  9. View Interpolation (Chen and Williams, 1993) • Input: A set of synthetic images with known depth and camera parameters (location, focal length, etc) • Computes optical flow maps related each pair of images • Optical flow map is the set of vectors describing where each point in the first image moves to in the second image • Morphs between images by moving points along flow vectors • Intermediate views are “real” views only in special cases

  10. View Morphing (Seitz and Dyer, 1997) • Uses interpolation to generate new views such that the intermediate views represent real camera motion • Observation: Interpolation gives correct intermediate views if the initial and final images are parallel views

  11. View Morphing Process • Basic algorithm: • Pre-warp input images to get them into parallel views • Interpolate for intermediate view • Post-warp to get final result • User specifies a camera path that rotates and translates initial camera onto final camera • Requires knowledge of projection matrices for the input images • Found with vision algorithms. User may supply correspondences • Intermediate motion can be specified by giving trajectories of four points

  12. Plenoptic Modeling (McMillan and Bishop, 1995) • Input: Set of panoramic images gathered by panning a video camera about its vertical optical axis on a tripod • The algorithm: • Determines the properties of each camera and registers the images associated with each pan • Determines the relative locations of each camera • Determines the depth of each point seen by multiple cameras • Generates new views by reconstructing the plenoptic function from the available samples

  13. Fitting Each Camera • Problem: • Given several images all taken with the same camera rotated about its optical center • Determine the camera parameters • Determine the angle between each image • Approach: • Multi-stage optimization • First stage estimates angle between images and focal length • Second stage refines and gets remaining parameters • When done, can map a pixel in one image to its correct position in any other images

  14. Locating the Cameras • Given cylindrical projections (panoramic images) for two cameras • User identifies 100-500 tie-points (points seen in both images) • Each tie-point defines two rays – one from the center of each camera through the tie-point • These rays should intersect at the world location for the point • Minimization step finds the camera locations and some other parameters that minimize the perpendicular distance between all the rays

  15. Determining Disparity • The minimization algorithm gives us the world location of the tie-points, but what about the rest of the points in the image? • Use standard computer vision techniques to find the remaining disparities • Disparity is the angular difference between the locations of a point in two images • Directly related to the depth of the point • Makes heavy use of the epipolar constraint

  16. The Epipolar Constraint • The location of a point in one image constrains it to lie on a ray in space passing through the camera and image point • This ray projects to a curve in the second image • Line for planar projection • Sine curve for cylindrical projection • Since the point must lie on the ray in world space, it must lie on the curve in the second image • Reduces the search for correspondences to a linear one along the line

  17. Reconstructing a New View • The disparity from a known cylinder pair can be transferred to a new cylinder, and then reprojected onto a plane (the image) • Can all be done in one step • Have to resolve occlusion problems • Two points from the reference image could map to the same point in the output image • Solution: Define a fill ordering that guarantees correct occlusion (an important contribution of this paper) • Also have to fill holes

  18. Summary • Image-based rendering obviously relies heavily on computer vision techniques • Particularly: Depth from stereo • The problem is very hard with real images • These techniques are not perfect! • Sampling remains a problem • Images tend to appear blurry • Relatively little work on reconstruction algorithms

More Related