1 / 40

Background Subtraction and Matting

Background Subtraction and Matting. © Yuri Bonder . 15-463: Computational Photography Alexei Efros, CMU, Fall 2005. “Smoke” (1996), the “photo album scene”. Moving in Time. Moving only in time, while not moving in space, has many advantages No need to find correspondences

lei
Download Presentation

Background Subtraction and Matting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Background Subtraction and Matting © Yuri Bonder 15-463: Computational Photography Alexei Efros, CMU, Fall 2005

  2. “Smoke” (1996), the “photo album scene”

  3. Moving in Time • Moving only in time, while not moving in space, has many advantages • No need to find correspondences • Can look at how each ray changes over time • In science, always good to change just one variable at a time • This approach has always interested artists (e.g. Monet) • Modern surveillance video camera is a great source of information • There are now many such WebCams now, some running for several years!

  4. Image Stack • As can look at video data as a spatio-temporal volume • If camera is stationary, each line through time corresponds to a single ray in space • We can look at how each ray behaves • What are interesting things to ask? 255 time 0 t

  5. Example

  6. Median Image Getting the right pixels Average image

  7. Webcams • Lots of cool potential projects • PCA, weather morphing, weather extrapolation, etc.

  8. Input Video

  9. Average Image What is happening?

  10. Figure-centric Representation

  11. 3 closest frames median images Context-based Image Correction Input sequence

  12. Average/Median Image • What can we do with this?

  13. Background Subtraction - =

  14. Crowd Synthesis (with Pooja Nath) • Do background subtraction in each frame • Find and record “blobs” • For synthesis, randomly sample the blobs, taking care not to overlap them

  15. Background Subtraction • A largely unsolved problem… One video frame Estimated background Difference Image Thresholded Foreground on blue

  16. How does Superman fly? • Super-human powers? • OR • Image Matting and Compositing?

  17. Image Compositing

  18. Compositing Procedure 1. Extract Sprites (e.g using Intelligent Scissors in Photoshop) 2. Blend them into the composite (in the right order) Composite by David Dewey

  19. Compositing: Two Issues Semi-transparent objects Pixels too large

  20. Solution: alpha channel • Add one more channel: • Image(R,G,B,alpha) Sprite! • Encodes transparency (or pixel coverage): • Alpha = 1: opaque object (complete coverage) • Alpha = 0: transparent object (no coverage) • 0<Alpha<1: semi-transparent (partial coverage) • Example: alpha = 0.7 Partial coverage or semi-transparency

  21. Multiple Alpha Blending • So far we assumed that one image (background) is opaque. • If blending semi-transparent sprites (the “A over B” operation): • Icomp = aaIa + (1-aa)abIb • acomp = aa + (1-aa)ab • Note: sometimes alpha is premultiplied: im(aR,aG,aB,a): • Icomp = Ia + (1-aa)Ib • (same for alpha!)

  22. “Pulling a Matte” • Problem Definition: • The separation of an image C into • A foreground object image Co, • a background image Cb, • and an alpha matte a • Co and a can then be used to composite the foreground object into a different image • Hard problem • Even if alpha is binary, this is hard to do automatically (background subtraction problem) • For movies/TV, manual segmentation of each frame is infeasible • Need to make a simplifying assumption…

  23. Blue Screen

  24. Blue Screen matting • Most common form of matting in TV studios & movies • Petros Vlahos invented blue screen matting in the 50s. His Ultimatte® is still the most popular equipment. He won an Oscar for lifetime achievement. • A form of background subtraction: • Need a known background • Compute alpha as SSD(C,Cb) > threshold • Or use Vlahos’ formula: a = 1-p1(B-p2G) • Hope that foreground object doesn’t look like background • no blue ties! • Why blue? • Why uniform?

  25. The Ultimatte p1 and p2

  26. Blue screen for superman?

  27. Semi-transparent mattes • What we really want is to obtain a true alpha matte, which involves semi-transparency • Alpha between 0 and 1

  28. Matting Problem: Mathematical Definition

  29. Why is general matting hard?

  30. Solution #1: No Blue!

  31. Solution #2: Gray or Flesh

  32. Triangulation Matting (Smith & Blinn) • How many equations? • How many unknowns? • Does the background need to constant color?

  33. The Algorithm

  34. Triangulation Matting Examples

  35. More Examples

  36. More examples

  37. Removing Shadows (Weiss, 2001) • How does one detect (subtract away) shadows?

  38. Averaging Derivatives

  39. Recovering Shadows

  40. Compositing with Shadows

More Related