1 / 23

Boundary Matting for View Synthesis

2 nd Workshop on Image and Video Registration, July 2, 2004. Boundary Matting for View Synthesis. Samuel W. Hasinoff Sing Bing Kang Richard Szeliski. Dept. of Computer Science University of Toronto hasinoff@cs.toronto.edu. Interactive Visual Media Group Microsoft Research

harrietl
Download Presentation

Boundary Matting for View Synthesis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 2nd Workshop onImage and Video Registration, July 2, 2004 Boundary Matting for View Synthesis Samuel W. Hasinoff Sing Bing Kang Richard Szeliski Dept. of Computer Science University of Toronto hasinoff@cs.toronto.edu Interactive Visual Media Group Microsoft Research {sbkang,szeliski}@microsoft.com

  2. Motivation Superior view synthesis & 3D editing from N-view stereo • Two major limitations – even with perfect stereo! • Resampling blur • Boundary artifacts • Key approach: occlusion boundaries as 3D curves • More suitable for view synthesis • Boundaries estimated to sub-pixel

  3. Extension to stereo • Lambertian assumption B1 B2 B3 F Matting from Stereo • underdetermined Matting problem: Unmix the foreground & background • Triangulation matting • (Smith & Blinn, 1996) • multiple backgrounds • fixed viewpoint &object B3 B1 B2 F

  4. Model boundaries as 3D splines (currently linear) Assumptions boundaries are relatively sharp relatively large-scale objects no internal transparency Occlusion Boundaries in 3D 3D world view 2 (reference) view 1 view 3

  5. Geometric View of Alpha • alpha depends only on projected 3D curve, x • integration over each pixel F B pixel j alpha  partial pixel coverage on F side simulate blurring by convolving with 2D Gaussian

  6. Related Work • Natural image matting[Chuang et al., 2001] • based on color statistics - single image - user-assisted • Intelligent scissors[Mortenson, 2000] • geometric view of alpha

  7. Related Work • Bayesian Layer estimation[Wexler and Fitzgibbon, 2002] • matting from multiple images using triangulation + priors • - requires very high-quality stereo • alpha calculated at pixel level, only for reference • not suitable for view synthesis

  8. find occlusion boundary in reference view backproject to 3D using stereo depth project to other views initial guess for Bi and F optimize matting Boundary Matting Algorithm 3D world optimize view 1 view 2 (reference) view 3

  9. Initial Boundaries From Stereo • Find depth discontinuities • Greedily segment longest four-connected curves • Spline control points evenly spaced along curve • Tweak - snap to strongest nearby edge

  10. Use stereo to grab corresponding background-depth pixels from nearby views (if possible) Color consistency check to avoid mixed pixels occluded Background Estimation B1 B2 B3 F

  11. Foreground Estimation • Invert matting equation, given 3D curve and B • Aggregate F estimates over all views

  12. Optimization • Objective: Minimize inconsistency with matting over curve parameters, x, and foreground colors, F • Pixels with unknown B not included • Non-linear least squares, using forward differencing for Jacobian

  13. Additional Penalty Terms • Favor control points at strong edges • define potential field around each edgel • Discourage large motions (>2 pixels) • helps avoid degenerate curves

  14. Naïve object insertion (no matting)

  15. Object insertion with Boundary Matting

  16. Naïve object insertion (no matting) Object insertion with Boundary Matting

  17. Naïve object insertion (no matting) Object insertion with Boundary Matting boundaries calculated with subpixel accuracy

  18. Samsung commercial sequence

  19. Naïve object insertion (no matting) Object insertion with Boundary Matting

  20. Boundary Matting Naïve method

  21. Boundary Matting Naïve method

  22. Synthetic Noise boundary matting (sigma = 13) boundary matting (sigma = 26) background composite no matting boundary matting

  23. Concluding Remarks • Boundary Matting • better view synthesis • refines stereo at occlusion boundaries • subpixel boundary estimation • Future work • incorporate color statistics • extend to dynamic setting

More Related