Image Compositing and Matting

1. Image Compositing and Matting

2. Introduction • Matting and compositing are important operations in the production of special effects. These techniques enable directors to embed actors in a world that exists only in imagination, or to revive creatures that have been extinct for millions of years. • During matting, foreground elements are extracted from a film or video sequence. • During compositing, the extracted foreground elements are placed over novel background images.

3. Traditional approaches to matting • Traditional approaches to matting include blue-screen matting and rotoscoping. • The former requires filming in front of an expensive blue screen under carefully controlled lighting • The latter demands talent and intensive user interaction.

4. Rotoscoping • Rotoscoping --- the process of tracking contours in a video sequence

5. Basic problem of blue screen matting • Given an image of a foreground object shot in front of a backing color (blue screen or green screen or other colors) • Obtain a matte of the foreground object so that the foreground object can be blended into a new background image using the matte to produce a new composite image.

6. Notations • Let C = [R, G,B] denote a color with 0 ≤ R, G, B ≤ 1. • Let a denotes a transparency value with 0 ≤ a ≤ 1. • Foreground image color: Cf = [Rf ,Gf ,Bf ], af = 1 • Backing (screen) color: Ck = [0, 0,Bk], ak = 1 (assuming blue screen) • Original foreground object color: Co = [Ro,Go,Bo] • Background image color: Cb = [Rb,Gb,Bb], ab = 1 • Composite image color: Cc = [Rc,Gc,Bc]

7. Problem Statement • Given Cf and Cb at corresponding pixels, and Ck a known backing color, and assuming Cf = aoCo + (1 − ao)Ck • Determine ao and Co, which then gives the composite color Cc = aoCo + (1 − ao)Cb at the corresponding point, for all points that Cf and Cb share in common.

8. Solution • Since Cf = aoCo + (1 − ao)Ck , we have: • 4 unknowns, 3 equations… Rf = aoRo Gf = aoGo Bf = aoBo + (1 − ao)Bk .

9. Case 1: No Blue • There is no blue in Co, i.e., Bo=0, and Bk≠0. • Then • This case is very restrictive. • It rules out many colors, including grays because grays have blue.

10. Case 2: Gray and Skin Color • Assume Ro=aBo (or Go=aBo) and Bk≠0 • Then where

11. Example 1: Gray • Then • This case applies to science fiction movie in which the spaceships are mostly gray.

12. Example 2: Skin Color • This case applies to human faces, hands, legs, etc.

13. Solution for General Cases • To solve the matting problem in general: • Need to take the same image with two different backing colors. • This gives four equations for solving the four unknowns • Case 1: Use Two Different Shades of Blue. • Case 2: Use Two Different Backing Colors.

14. Case 1 • Use two different shades of blue Bk1 and Bk2 as backing colors. • Then • so

15. Case 2 • Use two different backing colors Ck1 and Ck2. • Or • Over-constrained, 1 unknowns, three equations

16. Case 2: Solution 1 • Adding up 3 equations:

17. Case 2: Solution 2 • Apply least squares method • Define: • Then • Least-squares

18. Least-Squares • Set

19. Case 1 Example

20. Case 2 Example

21. Other Developments in Matting • Matting Without Blue Screen • A method proposed by Ruzon and Tomasi • User specify object region and boundary region. • Alpha value of object region is set to 1. • Alpha value of boundary region is computed by estimating the contributions of neighboring objects’ colors.

22. Example

23. Bayesian Approach

24. Shadow Matting • Pull a matte of shadow. • Acquire photometric and geometric properties of the target scene by sweeping oriented linear shadows across it. • Then, composite the shadow onto the scene.

25. Example

26. Environmental Matting Left: alpha matte. Middle: environment matte. Right: photo.

27. References • Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski. A bayesian approach to digital matting. In Proc. IEEE CVPR, pages II–264–II–271, 2001. • Y.-Y. Chuang, D. B. Goldman, B. Curless, D. H. Salesin, and R. Szeliski. Shadow matting and compositing. ACM Transactions on Graphics, 22(3):494–500, July 2003. • M. A. Ruzon and C. Tomasi. Alpha estimation in natural images. In Proc. IEEE CVPR, pages 18–25, 2000. • A. R. Smith and J. F. Blinn. Blue screen matting. In Proc. ACM SIGGRAPH, pages 259–268, 1996. • D. E. Zongker, D. M. Werner, B. Curless, and D. H. Salesin. Environment matting and compositing. In Proc. SIGGRAPH, pages 205–214, 1999.

28. Digital Compositing • Digital compositing means “digitally manipulated integration of at least two source images to produce a new image.” • The new image must appear realistic. • It must be completely and seamlessly integrated, as if it were actually photographed by a single camera.

29. Example 1

30. Example 2

31. More examples • http://www.beezlebugbit.com/digital/efx/efx_top.htm

32. Main Topics • Alpha blending: blending foreground and background • Keying: separating foreground and background • Luma, chroma, difference keying • Rig removal: removing unwanted elements

33. Alpha Blending • C = [α F + (1 – α) B] • If α = 1, then C = F, foreground is shown, i.e., foreground is opaque. • If α = 0, then C = B, background is shown, i.e., foreground is transparent. • 0 < α < 1: semi-transparent, e.g., shadow, smoke, etc. • If α ranges from 0 to 255, then the formula becomes: C = [α F + (1 – α) B] / 255

34. Example: No Background

35. Example: With Background

37. Note • For shadow, a must take fractional value (0 < α < 1). Otherwise, shadow looks unreal.

38. Boundary area • a at boundary area should also be fractional. Otherwise, have dark fringes; unrealistic.

39. Summary • A good matte has fractional a in shadow, and along object boundaries and shadow boundaries.

40. Keying • Separating foreground from background, creating a matte of foreground. • Also called pulling a matte (of foreground), or keying out (i.e., making transparent) background. • Recall: • A good matte has fractional a in shadow, and along object boundaries and shadow boundaries.

41. Basic methods • Luma keying: based on luminance (i.e., intensity) • Chroma keying: based on color (i.e., blue screen, green screen) • Difference keying: requires a clean plate, i.e., a background image without the foreground element.

42. Basic Idea • Compute difference between foreground and background (based on luma, chroma, or color) • Very small diff a = 0. • Very large diff  a = 1. • Intermediate diff  intermediate a

43. Luma Keying • Key out the background based on luminance. • Useful when background has a uniform luminance that is very different from foreground luminance.

44. Result

45. Chroma Keying • Key out the background based on color. • Useful when background has a uniform color that is very different from foreground color. • Example: Image shot with blue screen.

46. Characteristics of blue screen image

47. Difference Keying • More general than luma and chroma keying. • Key out background based on pixel-wise color difference between foreground and background footage.

48. Final Composition

49. Rig Removals • Rigs are equipment that support the actors or the props. • Sometimes, rigs cannot be removed by keying alone. • So, have to apply masking technique to remove rigs. • Need clean plate of background footage. • If camera moves, then need motion-controlled camera: • Computer controls camera to move the same way twice: • Without foreground objects; get clean plate. • With foreground objects.

50. Basic Idea • Apply a mask to mask out the rig. • Then, replace pixels in masked area by corresponding pixels in clean plate background. • If rig moves in footage, then have to animate the mask accordingly.