300 likes | 476 Views
Learning to Detect A Salient Object. Reporter: 鄭綱 (3/2). Outline. Introduction CRF Formulation of Static Salient Object Strong Contrast Center-Surround Histogram Color Spatial-Distribution Learning & Inference for the Model Formulation of Dynamic Salient Object Results Conclusion
E N D
Learning to Detect A Salient Object Reporter: 鄭綱(3/2)
Outline • Introduction • CRF • Formulation of Static Salient Object • Strong Contrast • Center-Surround Histogram • Color Spatial-Distribution • Learning & Inference for the Model • Formulation of Dynamic Salient Object • Results • Conclusion • References
Introduction • Image Labeling Problem Tree Sky Building Plane Lawn
Introduction • Which kinds of information can be used for labeling? • Features from individual sites • Intensity, color, texture, … Vegetation • Interactions with neighboring sites • Contextual information Sky or Building?
Introduction • Contextual information: 2 types of interactions • Interaction with neighboring labels (Spatial smoothness of labels) • neighboring sites tend to have similar labels(except at the discontinuities) Sky • Interactions with neighboring observed data Sky Building
Introduction • Let be the label of the node of the image set S, and Nibe the neighboring nodes of node i. • Three kinds of information for image labeling • Features from local node • Interaction with neighboring labels • Interaction with neighboring observed data node i S-{i} Ni
Introduction • General formulation: where and are called association potential and interaction potential.
Introduction Labels in Spatial Data are NOT independent! • spatially adjacent labels are often the same (Markov Random Fields and Conditional Random Fields) • spatially adjacent elements that have similar features often receive the same label (Conditional Random Fields) • spatially adjacent elements that have different features may not have correlated labels (Conditional Random Fields)
Formulation of Static Salient Object Detection • CRF model (static image): Maximize!! Minimize!! Salient object feature Pairwise feature Strong contrast Center-surround histogram Color spatial-distribution
Strong Contrast • Generate contrast map for each level of 6-level Gaussian pyramid. Then do linear combination. Level 4 Input image Level 1
Center-Surround Histogram • Salient object usually has a “huge” difference from local area. ……. More different between 2 rectangles !
Center-Surround Histogram where ?? …...
Color Spatial-Distribution • The wider a color is distributed in the image, the less possible a salient object contains this color. • Each pixel is assigned to a color component with the probability:
Color Spatial-Distribution • Then the feature can be defined as a weighted sum:
Formulation of Static Salient Object Detection • CRF model (static image): Maximize!! Minimize!! Salient object feature Pairwise feature Strong contrast Center-surround histogram Color spatial-distribution
Learning & Inference for the Model • The goal of CRF learning is to estimate the linear weights . Training images Gradient descent
Learning & Inference for the Model A training image for example: An training image Label inferred this iterate where Label per pixel Possible label
Learning & Inference for the Model Gradient descent: How to use ground-truth information? Where is the labeled ground-truth. t: iteration
Learning & Inference for the Model • Situations: Ground-truth mistake!!!
Ground-Truth Mistake • Solution: apply Gaussian function to weight every pixel in the rectangle.
Inference • We should find the most probable labeling to maximize in training & detection. • BP – Max-product Belief Propagation [Pearl ‘86] + Can be applied to any energy function – In vision results are usually worse than that of graph cuts – Does not always converge • TRW - Max-product Tree-reweighted Message Passing [Wainwright, Jaakkola, Willsky ‘02] , [Kolmogorov ‘05] + Can be applied to any energy function + Convergence guarantees for the algorithm in [Kolmogorov ’05]
Formulation of Dynamic Salient Object Detection • Similar as static salient object detection! Maximize!! Penalty term of motion Static Salient feature Contrast of motion Center-surround histogram Spatial-distribution of motion
Results From left to right: input image, multi-scale contrast, center-surround histogram, color spatial distribution, and binary mask by CRF.
Results From left to right: Fuzzy growing based method Salience map Their approach Ground-truth
Results Image set A Image set B 1. multi-scale contrast 2. center-surround histogram 3. color spatial distribution 4. combination all 1. FG 2. SM 3. their approach
Conclusion • They model the salient object detection by CRF, where a group of salient features are combined through CRF learning. • It’s a set of novel local, regional & global salient features to define a generic salient object. • Multi-object & no object cases are left as future work.
References (paper & book) • “Learning to Detect A Salient Object”, CVPR 2007, PAMI 2010. • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.4387&rep=rep1&type=pdf • “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis”, PAMI, 1998. • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.65.2199&rep=rep1&type=pdf • “Pattern Recognition and Machine Learning”, C. M. Bishop. • http://www.library.wisc.edu/selectedtocs/bg0137.pdf
References (about CRF) • “Discriminative Random Fields”, IJCV, 2006. • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.116.4695&rep=rep1&type=pdf • “Conditional Random Fields: An Introduction”, H. M. Wallach. • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.64.436&rep=rep1&type=pdf • “Log-linear models & conditional random fields”, C. Elkan, 2008. • http://cseweb.ucsd.edu/~elkan/250B/cikmtutorial.pdf
References (about TRW) • “MAP estimation via agreement on (hyper) trees: Message-passing and linear programming approaches”, IEEE transaction on Information Theory, 2005. • http://arxiv.org/pdf/cs/0508070 • “Convergent Tree-Reweighted Message Passing for Energy Minimization”, PAMI, 2006. • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.100.2409&rep=rep1&type=pdf • “A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors”, PAMI, 2008. • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.142.4997&rep=rep1&type=pdf