Introduction of Saliency Map

Introduction of Saliency Map Presenter: Chien-Chi Chen Advisor: Jian-Jiun Ding

Outline • Introduction of saliency map • Button-up approach • L. Itti’s approach • Frequency-tuned • Multi-scale contrast • Depth of field • Spectral Residual approach • Global contrast based • Top-down approach • Context-aware • Information maximum • Measuring visual saliency by site entropy rate

Introduction of saliency map • Low-level(contrast) • Color • Orientation • Size • Motion • Depth • High-level • People • Context Important! Low-level With face detection Judd et al, 2009

Outline • Introduction of saliency map • Button-up approach • L. Itti’s approach • Frequency-tuned • Multi-scale contrast • Depth of field • Spectral Residual approach • Global contrast based • Top-down approach • Context-aware • Information maximum • Measuring visual saliency by site entropy rate

L. Itti’s approach • Architecture: Gaussian Pyramids R,G,B,Y Gabor pyramids for q = {0º, 45º, 90º, 135º}

L. Itti’s approach • Center-surround Difference • Achieve center-surround difference through across-scale difference • Operated denoted by Q: Interpolation to finer scale and point-to-point subtraction • One pyramid for each channel: I(s), R(s), G(s), B(s), Y(s)where sÎ [0..8] is the scale

L. Itti’s approach • Center-surround Difference • Intensity Feature Maps • I(c, s) = | I(c)QI(s)| • cÎ {2, 3, 4} • s = c + d where dÎ {3, 4} • So I(2, 5) = | I(2) QI(5)|I(2, 6) = | I(2) QI(6)|I(3, 6) = | I(3) QI(6)| … •  6 Feature Maps

L. Itti’s approach Center-surround Difference Color Feature Maps Center-surround Difference Orientation Feature Maps Red-Green and Yellow-Blue Same c and s as with intensity +B-Y +Y-B +R-G +G-R +B-Y +G-R +Y-B +B-Y +R-G RG(c, s) = | (R(c) - G(c)) Q (G(s) - R(s)) | BY(c, s) = | (B(c) - Y(c)) Q (Y(s) - B(s)) |

L. Itti’s approach • Normalization Operator • Promotes maps with few strong peaks • Surpresses maps with many comparable peaks • Normalization of map to range [0…M] • Compute average m of all local maxima • Find the global maximum M • Multiply the map by (M – m)2

L. Itti’s approach Example of Operation: Inhibition of return

Frequency-tuned Image Average Gaussian blur

Multi-scale contrast • Saliency algorithm Multi-scale contrast Center-surround histogram Conditional Random Field Image Saliency map Color spatial-distribution

Multi-scale contrast Multi-scale contrast Center-surround histogram Distance between histograms of RGB color: • Local summation of laplacian pyramid

Multi-scale contrast • Color spatial-distribution The variance of Coordinate of pixel x and y Image(RGB) GMM Distance from pixel x to image center

Multi-scale contrast • Energy term: • Saliency object: • Pairwise feature:

Multi-scale contrast • CRF: • The derivative of the log-likelihood with respect to

Depth of field • As the spread of single lens reflex camera, more and more low depth of field(DOF) images are captured. • However, current saliency detection methods work poorly for the low DOF images.

Depth of field • Algorithm:

Depth of field • Classification: • Focal Point: In a low DOF image Rectangle with the highest edge density, and center is initial focal point DOG • Composition Analysis: segmentation Region

Spectral Residual Approach • First scaling image to 64x64. • Then we smoothed the saliency map with a gaussian filter g(x) ( = 8).

Global contrast-based • Histogram based contrast(Lab): Quantization of Lab Each channel to have 12 different value 85

Global contrast-based • Region based contrast • Segment the Image • [Efficient graph-based image segmentation]

Outline • Introduction of saliency map • Button-up approach • L. Itti’s approach • Frequency-tuned • Center-surround • Depth of field • Spectral Residual approach • Global contrast based • Top-down approach • Context-aware • Information maximum • Measuring visual saliency by site entropy rate

Context-Aware • Goal: Convey the image content Liu et al, 2007

Context-Aware • Distance between a pair of patches: High salient

Context-Aware • Distance between a pair of patches: K most similar patches at scale r High for K most similar Saliency

Context-Aware • Salient at: • Multiple scales  foreground • Few scales  background Scale 1 Scale 4

Context-Aware • Foci = • Include distance map X

Outline • Introduction of saliency map • Button-up approach • L. Itti’s approach • Frequency-tuned • Center-surround • Depth of field • Spectral Residual approach • Global contrast based • Top-down approach • Context-aware • Information maximum • Measuring visual saliency by site entropy rate

Measuring visual saliency by site entropy rate 1

Measuring visual saliency by site entropy rate 2 A fully-connected graph representation is adopted for each

Sub-band graph representation

Measuring visual saliency by site entropy rate 3 A random walk is adopted on each sub-band graph. And Site entropy rate(SER) is measured the average information from a node to the other

The site entropy rate

Conclusion • Image processing is funny • Unusual in its neighborhood will correspond to high saliency weight • Contrast is the key of saliency

Reference [1] R. Achanta, F. Estrada, P. Wils, and S. S¨usstrunk. Salient region detection and segmentation. In ICVS, pages 66–75. Springer, 2008. 410, 412, 414 [2] R. Achanta, S. Hemami, F. Estrada, and S. S¨usstrunk. Frequency-tuned salient region detection. In CVPR, pages 1597–1604, 2009. 409, 410, 412, 413, 414, 415 [3] L. Itti, C. Koch, and E. Niebur. A model of saliency based visual attention for rapid scene analysis. IEEE TPAMI, 20(11):1254–1259, 1998. 409, 410, 412, 414 [4] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR, pages 1–8, 2007. 410, 412, 413, 414 [5] S. Goferman, L. Zelnik-Manor, and A. Tal. Context-aware saliency detection. In CVPR, 2010. 410, 412, 413, 414, 415 [6] MM Cheng, GX Zhang, N. J. Mitra, X. Huang, S.M. Hu. Global Contrast based Salient Region Detect. In CVPR, 2011 . [7] T. Liu, Z. Yuan, J. Sun, J.Wang, N. Zheng, T. X., and S. H.Y. Learning to detect a salient object. IEEE TPAMI, 33(2):353–367, 2011. 410 [8] W. Wang, Y. Wang, Q. Huang, W. Gao, Measuring Visaul Saliency by Site Entropy Rate, In CVPR, 2010.

Introduction of Saliency Map