1 / 50

Image-based rendering

Image-based rendering. Michael F. Cohen Microsoft Research. Computer Graphics. Output. Image. Model. Synthetic Camera. Computer Vision. Output. Model. Real Scene. Real Cameras. Combined. Output. Image. Model. Real Scene. Synthetic Camera. Real Cameras.

shaman
Download Presentation

Image-based rendering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image-based rendering Michael F. Cohen Microsoft Research

  2. Computer Graphics Output Image Model Synthetic Camera

  3. Computer Vision Output Model Real Scene Real Cameras

  4. Combined Output Image Model Real Scene Synthetic Camera Real Cameras

  5. But, vision technology falls short Output Image Model Synthetic Camera Real Scene Real Cameras

  6. … and so does graphics. Output Image Model Synthetic Camera Real Scene Real Cameras

  7. Image Based Rendering Output Image Synthetic Camera Real Scene Images+Model Real Cameras -or- Expensive Image Synthesis

  8. Ray • Constant radiance • time is fixed • 5D • 3D position • 2D direction

  9. All Rays • Plenoptic Function • all possible images • too much stuff!

  10. Line • Infinite line • 4D • 2D direction • 2D position

  11. Ray • Discretize • Distance between 2 rays • Which is closer together?

  12. Image • What is an image? • All rays through a point • Panorama?

  13. Image • 2D • position of rays has been fixed • direction remains

  14. Image • Image plane • 2D • position

  15. Image • Image plane • 2D • position

  16. Object • Light leaving towards “eye” • 2D • just dual of image

  17. Object • All light leaving object

  18. Object • 4D • 2D position • 2D direction

  19. Object • All images

  20. Lumigraph • How to • organize • capture • render

  21. Lumigraph - Organization • 2D position • 2D direction s q

  22. Lumigraph - Organization • 2D position • 2D position • 2 plane parameterization u s

  23. s,t u,v s,t u,v Lumigraph - Organization • 2D position • 2D position • 2 plane parameterization t v u s

  24. Lumigraph - Organization • Hold s,t constant • Let u,v vary • An image s,t u,v

  25. Lumigraph - Organization • Discretization • higher res near object • if diffuse • captures texture • lower res away • captures directions s,t u,v

  26. Lumigraph - Capture • Idea 1 • Move camera carefully over s,t plane • Gantry • see Lightfield paper s,t u,v

  27. Lumigraph - Capture • Idea 2 • Move camera anywhere • Rebinning • see Lumigraph paper s,t u,v

  28. Lumigraph - Rendering • For each output pixel • determine s,t,u,v • either • find closest discrete RGB • interpolate near values s,t u,v

  29. s u Lumigraph - Rendering • For each output pixel • determine s,t,u,v • either • use closest discrete RGB • interpolate near values

  30. s u Lumigraph - Rendering • Nearest • closest s • closest u • draw it • Blend 16 nearest • quadrilinear interpolation

  31. High-Quality Video View Interpolation Using a Layered Representation Larry Zitnick Sing Bing Kang Matt Uyttendaele Simon Winder Rick Szeliski Interactive Visual Media Group Microsoft Research

  32. Current practice free viewpoint video Many cameras vs. Motion Jitter

  33. Current practice free viewpoint video Many cameras vs. Motion Jitter

  34. Video view interpolation Fewer cameras and Smooth Motion Automatic Real-time rendering

  35. Prior work: IBR (static) Plenoptic Modeling McMillan & Bishop, SIGGRAPH ‘95 Light Field Rendering Levoy & Hanrahan, SIGGRAPH ‘96 Concentric Mosaics Shum & He, SIGGRAPH ‘99 The Lumigraph Gortler et al., SIGGRAPH ‘96

  36. Prior work: IBR (dynamic) Stanford Multi-Camera Array Project Dynamic Light Fields Goldlucke et al., VMV ‘02 Virtualized RealityTM Kanade et al., IEEE Multimedia ‘97 3D TV Matusik & Pfister, SIGGRAPH ‘04 Image-Based Visual Hulls Matusik et al., SIGGRAPH ‘00 Free-viewpoint Video of Humans Carranza et al., SIGGRAPH ‘03

  37. System overview Video Capture Video Capture OFFLINE Compression Representation Stereo File ONLINE Selective Decompression Render

  38. cameras cameras hard disks controlling laptop concentrators

  39. Calibration Zhengyou Zhang, 2000

  40. Input videos

  41. Key to view interpolation: Geometry Stereo Geometry Image 1 Image 2 Camera 1 Camera 2 Virtual Camera

  42. Good Bad Match Score Match Score Match Score Match Score Image correspondence Image 1 Image 2 Leg Correct Incorrect Wall

  43. Image 1 Image 2 Local matching Low texture

  44. Image 2 Image 1 E B A F C D Global regularization A Create MRF (Markov Random Field): R Q P A S T U Number of states = number of depth levels colorA ≈ colorB → zA ≈ zB zA ≈ zP, zQ, zS Each segment is a node

  45. Iteratively solve MRF

  46. Depth through time

  47. Matting Background Surface Interpolated view without matting Foreground Surface Background Background Alpha Strip Width Foreground Foreground Bayesian Matting Chuang et al. 2001 Camera

  48. Rendering with matting Matting No Matting

  49. Boundary Layer: Color Alpha Depth Representation Main Background Boundary Strip Width Foreground Main Layer: Color Depth

  50. “Massive Arabesque” videoclip

More Related