1 / 46

Layered Depth Images

Layered Depth Images. Jonathan Shade University of Washington Steven Gortler Harvard University Li-wei He Stanford University Richard Szeliski Microsoft research Presented by Chung, Shuo-Heng. Introduction.

sage
Download Presentation

Layered Depth Images

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Layered Depth Images Jonathan Shade University of Washington Steven Gortler Harvard University Li-wei He Stanford University Richard Szeliski Microsoft research Presented by Chung, Shuo-Heng

  2. Introduction The most familiar Image Base Rendering method is texture mapping, however… 1.Aliasing. e.q. Infinite checkerboard 2.Speed is limited by the surface the texture applied. e.q. Tree with thousands of leafs

  3. Introduction • Two extensions have been presented to address these two difficulties: • 1.Sprite • If the new view is near the original view, the new image can be created by altering the original sprite • cannot produce parallax well • 2.Depth Image • gap will be introduced due to visibility changes when some portion of the scene become unoccluded or a surface is magnified.

  4. Introduction • The paper introduces two extension: • sprite with depth • layered depth image.

  5. Previous Work • Max: • use a representation similar to an LDI. • Focus on high-quality anti-aliasing. • Warping multiple input LDI with different camera. • Mark et al. and Darsa et al: • create triangulated depth maps from input images with per-pixel depth. • Taking advantage of graphics hardware pipelines.

  6. Previous Work • Shade et al. and Shaufler et al.: • Render complex portions of a scene onto a sprites. • Reuse these sprites onto in subsequent frames. • Lengyel and Snyder extended above work: • Apply affine transformation to fit a set of sample points. • Transforms are allowed to change as sample points change. • Horry et al.: • Use a single input and some user supplied information. • Be able to provide approximate 3D cues.

  7. Previous Work McMillan’s ordering algorithm:

  8. Sprites Texture maps or images with alphas rendered onto planar surface. 1.Used directly as drawing primitives. 2.Used to generate new views by warping.

  9. Sprites • Where is pixel x,y from image 1 located in image 2? • Assume image 1 a picture of the x-y coordinate plane at z = 0 • Plane can be placed arbitrarily, but setting z = 0 allows us to ignore z coordinate • C1 = WPV, takes point on plane (x,y) to pixel (x1,y1) x1,y1 x,y cam1 cam2 z = 0 x2,y2

  10. Sprites w = w2 / w1

  11. Sprites with Depth epipole

  12. Sprites with Depth Forward map has a problem. When the view suddenly changes, gap may be introduced.

  13. Sprites with Depth In order to deal with the problem, following steps are used: 1.Forward map the displacement map d1(x1, y1) to get d3(x3, y3) => d3(x3, y3) = d1(x1, y1) (x1, y1) (x3, y3) H1,2 is obtained by dropping the third row and column of C2C1 e1,2 is the third column of H1,2 -1

  14. Sprites with Depth 2.Backward map d3(x3, y3) to obtain d2(x2, y2) d2(x2, y2) = d3(x3, y3)

  15. Sprites with Depth 3.Backward map the original sprite color. Assign the color in input image (x1, y1) to output image (x2, y2)

  16. Sprites with Depth • These steps (first forward mapping the displacement and then backward mapping with the new displacements) have following advantages: • Small errors in displacement map warping will not be as evident as errors in the sprite image warping • We can design the forward warping step to have a simpler form by factoring out the planar perspective warp.

  17. Sprites with Depth We can rewrite the equation:

  18. Sprites with Depth Another faster but less accurate variance: In the first step, set u3(x3, y3) = x1 – x3 and v3(x3, y3) = y1 – y3 In the third step, we use instead of

  19. Sprites with Depth Warped with both three steps Warped with homography and crude parallax (d1) Warped with homography and true parallax (d2) Warped by homography only (no parallax) Input color (sprite) image Input depth map d1(x1, y1) Pure parallax warped depth map d3(x3, y3) Forward warped depth map d2(x2, y2) Forward warped depth map without parallax correction Sprite with “pyramid” depth map

  20. Recovering sprite from image sequences • We can use computer vision technique to extract sprite from image sequences.. • Segment the sequence into coherently moving regions with a layered motion estimation algorithm. • Compute a parametric motion estimate (planar perspective transformation) for each layer. • Determine the plane equation associated with each region by tracking feature points from frame to frame.

  21. Recovering sprite from image sequences Initial segmentation into six layers The third of five images Recovered depth map

  22. Recovering sprite from image sequences The five layer sprites Residual depth image for fifth layer

  23. Recovering sprite from image sequences Re-synthesized third image Novel view without residual depth Novel view with residual depth

  24. Layered Depth Image Layered depth image can handle more general disocclusions and large amount of parallax as the viewpoint moves. There are three ways presented in the paper to construct an LDI. 1. LDIs from multiple depth image. 2. LDIs from a modofied ray tracer. 3. LDIs from real images.

  25. Layered Depth Image The structure of an LDI: DepthPixel = ColorRGBA: 32 bit integer Z: 20 bit integer SplatIndex: 11 bit integer LayeredDepthPixel = NumLayers: integer Layers[0..numlayers-1]: array of DepthPixel LayeredDepthImage = Camera: camera Pixels[0..xres-1,0..yres-1]: array of LayeredDepthPixel

  26. LDI from Multiple Depth Images LDI can be constructed by warping n depth image into a common camera view.

  27. LDI from a Modified Ray Tracer Ray can be cast from any point on cue face A to any point on frustum face B. B A

  28. LDI from a Modified Ray Tracer • “Uniformly” Sample the scene. • Any object intersection hit by the ray is reprojected into the LDI. • If the new sample is within a tolerance in depth of an exsiting depth pixel, the new sample color is averaged with the existing depth pixel. Otherwise a new depth pixel is created.

  29. LDI from Real Images Use voxel coloring algorithm to obtain the LDI directly from input images.

  30. LDI from Real Images This is a dinosaur model reconstructed from 21 photographs.

  31. Space Efficient Representation • It is important to maintain the spatial locality of depth pixels to exploit the cache in CPU. • Reorganize the depth pixel into a linear array ordered from bottom to top and left to right in screen space, and back to front along the ray. • A accumulated depth array is maintained in each scanline and the depth can be retrieved with the layer number as index.

  32. Incremental Warping Computation Recall that:

  33. Incremental Warping Computation

  34. Incremental Warping Computation We can simply increment the start to get the warped position of the next layered depth pixel along a scanline.

  35. Splat Size Computation To splat the LDI into the output image, we roughly approximate the projected area of the warped pixel. res1 = 1/(w1h1) res2 = 1/(w2h2) LDI camera Output camera

  36. Splat Size Computation The square root size can be approximated more efficiently.

  37. Splat Size Computation The square root size can be further approximated by a lookup table. 5 bits for d1 6 bits for normal (3 bits for nx, 3bits for ny) Total 11 bits => 2048 possible values.

  38. Splat Size Computation The paper uses four splat sizes: 1x1, 3x3, 5x5 and 7x7. Each pixel in a footprint has an alpha value to approximate a Gaussian splat kernel. These alpha value are rounded to 1, ½, or ¼, so the alpha blending can be done with integer shift and add.

  39. Depth Pixel Representation • To fit four depth pixels into a single cache line (32 bytes in Pentiumm Pro and Pentium II) • Convert the floating point Z value into a 20 bit integer. • Splat table index size is 11 bits. • R, G, B, and alpha values fill out the other 4 bytes. • This yields a 25 percent improvement in rendering speed.

  40. Clipping • Split the LDI frustum into two segments, a near and a far one. • Near and far segment are clipped individually. • The near frustum is kept smaller then the far segment. • Intersect the view frustum with the frustum of the LDI. • Only visible pixels are rendered. • Far segment first, then near segment. • Speed rendering time by a factor of 2 to 4.

  41. Clipping

  42. Warping summary

  43. Result

  44. Result

  45. Result

  46. Future Work • To explore representations and rendering algorithms combining several IBR techniques. • Automatic techniques for taking a 3D scene and re-represented it in the most appropriate fashion for image base rendering.

More Related