Image Matting and Its Applications

1. Image Matting and Its Applications Chen-Yu Tseng Advisor: Sheng-Jyh Wang 2012-10-29

2. Image Matting • A process to extract foreground objects from an image, along with an alpha matte (the opacity of the foreground color) Input Image Alpha Matte Extracted Foreground

3. Two Approaches of Image Matting • Supervised Matting • With User’s Guidance • Unsupervised Matting • Without User’s Guidance Input Image User’s Guidance e.g. Trimap: White  Foreground Black  Background Unknown  Gray

4. Two Schemes of Supervised Matting Propagation-based Scheme Sampling-based Scheme Infer Alpha Matte with Some Color Samples A Local-based Approach • Infer Alpha Matte with Propagation through a Graphical Model • A Global-based Approach • Unknown • Pixel • Foreground • Color Set Foreground Pixel • Background • Color Set • Unknown Pixel • Background Pixel

5. Propagation-based scheme - Matting Laplacian Approach • A Graphical Model with Connectivity between Pixels • The Connectivity Is Learned from the Image Structure • Capability for Dealing with Both • Supervised Matting (Inference Problem) • Unsupervised Matting (Decomposition Problem) Foreground Pixel • Unknown Pixel • Background Pixel

6. Reference of Matting LaplacianApproach • First proposed by Levin et al. for supervised matting (closed-form matting) • A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008. • Extended to unsupervised matting (spectral matting) • A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no. 10, pp. 1699-1712, Oct. 2008. • Extended to learning-based matting • Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In ICCV, pages 889–896, 2009. • Extended to multi-layer matting • D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,” IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.

7. Matting Laplacian Background Estimating Pair-wise Affinity Supervised Matting Input Image Graphical Model Foreground Node: Image Pixels Edge: Affinity Matting Laplacian Matrix: Recording the Connectivity between Pair of Pixels

8. Introduction of Graph Laplacian • Vertex Index • : Adjacency Matrix 2 1 : LaplacianMatrix 3 4 5 : Degree Matrix • : Adjacency Matrix A Graph with Five Vertexes

9. Cutting Cost Function with Graph Laplacian Cost Function for Cutting Criterion Low-cost Assignment High-cost Assignment 2 1 2 1 3 4 3 4 5 5

10. Construction of Matting Laplacian • Color-model-based Approach (Original) • Estimating Affinity Based on Relative Color Distance • Learning-based Approach (Extended) • Learning Affinity Based on Image Structure

11. Construction of Matting LaplacianColor-model-based Approach b g r Color Distribution Input Image • A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.

12. Construction of Matting LaplacianLearning-based Approach • Learning Affinity among Local Pixels Extending to a Local Patch q Assuming all Pixels Sharing the Same Linear Coefficient Linear Alpha-color Model for Single Pixel: : Alpha Vector for Patch q : Feature Matrix : Linear Coefficient : Alpha Value for Pixel i : Feature Vector () : Linear Coefficient

13. Construction of Matting LaplacianLearning-based Approach Derived Linear Coefficient Rewritten Linear Model

14. Construction of Matting LaplacianLocal Cost Function Local Linear Model Local Cost Function Patch q : Local Laplacian Matrix for Patch q Input Image

15. Construction of Matting LaplacianLocal  Global Local Cost Function Patch q : Local Laplacian Matrix for Patch q Global Cost Function Input Image

16. Supervised Matting (Closed-form Matting) • Foreground Pixel User’s Guidance, Input Image • Unknown Pixel • Background Pixel Cost Function for Supervised Matting Optimal Solution Affinity Cost Data Cost

17. Experimental Results Input Image Alpha Matte Synthesized Result

18. Unsupervised Matting (Spectral Matting) • Solving Alpha Matte without User’s Guidance • Procedures • Decomposing Image into Several Matting Components • Combining Matting Components into Alpha Matte

19. Spectral Clustering s.t.=1 : Eigenvector : Eigenvalue L is symmetric and positive semi-definite. The smallest eigenvalue of L is 0, the corresponding eigenvector is the constant one vector 1. L has n non-negative, real-valued eigenvalues 0= λ1 ≦ λ 2 ≦ . . . ≦ λ n. 2 1 3 4 5 • : Laplacian Matrix A Graph Example

20. Spectral Clustering & Matting Components • Linear • Transformation • Zero-Eigenvectors • : Laplacian Matrix • Binary Indicating Vectors

21. Overview of Spectral Matting K-means Clustering & Linear Transformation Input Image Matting Laplacian Matting Components Smallest Eigenvectors

22. Spectral Clustering & K-means Pixel i Input Image s-dimensional Space K-means Clustering … s-smallest Eigenvectors

23. K-means Projection into Eigen Space Generating Matting Components … … … Smallest Eigenvectors

24. Reconstructing Alpha Matte from Matting Components Input Image Matting Components + + = Selected Matting Components Alpha Matte

25. Reconstructing Alpha Matte by Grouping Matting Components Alpha Matte Generation : Combination Vector Matting cost function Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte

26. Results by Levin et al.

27. Summary • Constructing Matting Laplacian • Solving Supervised Matting Problem • Solving Unsupervised Matting Problem

28. Proposed Approaches • Efficient Cell-based Framework for Reducing Computations • Multi-scale Analysis • Extended Applications (Depth Image Reconstruction) Depth Reconstruction in Shape From Focus (SFF) Depth Reconstruction from Single Image Reconstructed Depth Input Image Reconstructed Depth Input Image

29. Cell-based Framework Pixel-wise Affinity Conventional Matting Laplacian Pixel-wise Data Distribution Cell-wise Affinity Image Cell-based Matting Laplacian Cell-wise Data Distribution

30. Multi-scale Affinity Learning Cell-based Approach Image & Computation Patterns Pixel-based Approach

31. Multi-scale Affinity Learning … … Finest Level Coarsest Level Cell-based Graph

32. Results of Reconstructed Alpha Matte Input • (a) Grouping Results by Levin et al. • (b)Grouping Results by Levin et al. with Coarse-to-fine Scheme. (c) Ours 2nd Rank 1st Rank

33. Results (a) Input images (b) Levin’s result (c) Our result

34. Proposed Approaches • Efficient Cell-based Framework for Reducing Computations • Multi-scale Analysis • Extended Applications (Depth Image Reconstruction) Depth Reconstruction in Shape From Focus (SFF) Depth Reconstruction from Single Image Reconstructed Depth Input Image Reconstructed Depth Input Image

35. Depth Reconstruction in Shape From Focus (SFF) Optical Direction W2 W1 W2 Optical Direction Multi-focus Image Sequence Focus Value W1

36. Low-SNR Problem • Spatially Varying Precision • Low-texture  Low-SNR • Leading Noisy Result Low- precision High- precision Input Image Observation

37. Proposed Maximum-a-posteriori Estimation Local Learning Learning-based Graph Inference Multi-focus Image Sequence Reconstructed Depth

38. Proposed Maximum-a-posteriori Estimation : Optimal Result : Depth Image : Observation : Input Image Posterior Likelihood Prior Learned from Image Local Observation with Spatial-varying Precision

39. Likelihood Model Posterior Likelihood Prior Local Observation with Spatial-varying Precision Low- precision High- precision Input Observation Precision Result

40. Prior Model Posterior Likelihood Prior Learning from Input Image Local Learning Learning-based Graph Multi-focus Image Sequence

41. Maximum-a-posteriori Estimation for Depth Reconstruction Input Image Observation Reconstructed Depth

42. Results of Shape from Focus Input Image S. Nayar, 1994 Ours M. Mahmood, 2012 T. Aydin, 2008

43. Conclusions • Construction of Matting Laplacian • Conventional Approach • Multi-scale Cell-based Approach • Supervised Matting • Spectral Matting • Depth Reconstruction