池 晨 @ jdl.ac.cn 2010.8.27

1. Context-Aware Saliency Detection {Stas Goferman ,Lihi Zelnik-Manor, Ayellet Tal} @ee.technion.ac.il 池 晨@jdl.ac.cn 2010.8.27

2. Outline • Authors • Study Background • Abstract • Context-Aware Saliency Detection • Results • Conclusion • Appendix

3. Stas Goferman Collage Stas Goferman, Ayellet Tal, Lihi Zelnik-Manor, Puzzle-like collage, Computer Graphics Forum (EUROGRAPHICS) 2010. Saliency Stas Goferman, Ayellet Tal, Lihi Zelnik-Manor, Context-aware saliency detection,CVPR2010. Stas Goferman MSc@Technion(graduated)

4. Lihi Zelnik-Manor(1/3) Analyzing and inferring the content of video data and image collections for real applications • L. Zelnik-Manor and P. Perona  "Automating Joiners” The 5th international symposium on non-photorealistic animation and rendering, NPAR 2007. • L. Zelnik-Manor and M.Irani"Statistical Analysis of Dynamic Actions“, IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 28(9): 1530--1535, September 2006.  • L. Zelnik-Manor and M. Irani "On Single Sequence and Multi-Sequence Factorizations“,International Journal of Computer Vision (IJCV),   67(3): 313-326,  May 2006. • L. Zelnik-Manor, G. Peters and P.Perona "Squaring the Circle in Panoramas” Tenth IEEE International conference on Computer Vision 2005, Volume 2, pp. 1292-1299 (ICCV'05). (pdf) Lihi Zelnik-Manor Faculty member@EE department of technion

5. Lihi Zelnik-Manor(2/3) • L. Zelnik-Manor and P. Perona "Self-Tuning Spectral Clustering", Advances in Neural Information Processing Systems 17, pp. 1601-1608, 2005, (NIPS'04). • L. Zelnik-Manor and M. Irani, "Temporal Factorization Vs. Spatial Factorization". Proc. European Conference of Computer Vision, Volume 2, pages 434-445, Prague, Czech Republic, 2004 (ECCV'04). • L. Zelnik-Manor  and  M. Irani, "Degeneracies, Dependencies and their Implications in Multi-body and Multi-Sequence Factorizations"  IEEE Conference on Computer Vision and  Pattern Recognition, Vol. 2, pp. 287-93, June 2003 (CVPR'03) • L.Zelnik-Manor, M. Machline and M.Irani,  "Multi-Body Factorization With Uncertainty: Revisiting Motion Consistency “ International Journal of Computer Vision (IJCV) Special Issue: Vision and Modelling of Dynamic Scenes, Vol. 68, No. 1, pp. 27-41, June 2006. Lihi Zelnik-Manor Faculty member@EE department of technion

6. Lihi Zelnik-Manor(3/3) • L. Zelnik-Manor  and  M. Irani,  "Event-Based Analysis of Video “ IEEE Conference on Computer Vision and  Pattern Recognition, December 2001 (CVPR'01). • L. Zelnik-Manor  and  M. Irani,  "Multi-View Constraints on Homographies“ IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Vol. 24, No. 2, pp. 214-223, February 2002. • L. Zelnik-Manor and M.Irani, "Multi-Frame Estimation of Planar Motion”IEEE Trans. on Pattern Analysis and machine Intelligence (PAMI), Vol. 22, No. 10, pp. 1105-1116, October 2000. • L. Zelnik-Manor and  M. Irani, "Multi-View Subspace Constraints on Homographies“,IEEE Seventh International Conference on Computer Vision, Vol. 2, pp. 710-715, September 1999 (ICCV'99). • L. Zelnik-Manor  and  M. Irani, "Multi-Frame Alignment of Planes“,IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 151-156, June 1999 (CVPR'99) Lihi Zelnik-Manor Faculty member@EE department of technion

7. Ayellet Tal(1/3) Received Ph.D. from the Department of Computer Science, Princeton University. On the editorial advisory board of Computers&Graphics. On the program committees of SIGGRAPH Asia, SMI, Pacific graphics and Applications of Computer Vision in Archaeology(CVPR). Ayellet Tal Faculty member@EE department of technion

8. Ayellet Tal(2/3) • NPR and Archaeology • Saliency & Collages • Texture Mapping & Augmented Reality • Mesh Segmentation and Applications • Ayellet Tal,Mesh, Segmentation for CAD applications, ESDA - 9th Biennial ASME Conference on Engineering Systems Design and Analysis, 2008 • Shape-Based Retrieval of 3D Models • Ayellet Tal and Emanuel, Zuckerberger,Mesh Retrieval by Components ,International Conference on Computer Graphics Theory and Applications, 142-149, 2006. • Collision Detection and Visibility • Metamorphosis and Reconstruction • Ayellet Tal ,Gershon Elber, Image Morphing with Feature Preserving Texture, Computer Graphics Forum (Eurographics '99), September 1999. Ayellet Tal Faculty member@EE department of technion

9. Ayellet Tal(3/3) • Algorithm Visualization and Graph Drawing • Ayellet Tal and David P. Dobkin, Visualization of Geometric Algorithms, Visualization '94, October 1994, 149-155.IEEE Transactions on Visualization and Computer Graphics, 1(2): 194-204, 1995 • Ayellet Tal, Algorithm Animation Systems for Constrained Domains, Software Visualization, State-of-the-Art Survey, LNCS 2269. Stephan Diehl (ed.), Springer Verlag, 2002. • Animating nature • Data Structures • Education • Database and Distributed Systems • Ayellet Tal , Rafael Alonso, Commit Protocols for Externalized-Commit Heterogeneous Database Systems, Distributed and Parallel Databases,2(2): 209-234, 1994. Ayellet Tal Faculty member@EE department of technion

10. Outline • Authors • Study Background • Abstract • Context-Aware Saliency Detection • Results • Conclusion • Appendix

11. Saliency concentrating on fixation points From left to right: The Original Image, Saliency map, Real fixation map. ROC curves

12. Saliency concentrating on dominate objects From left to right: The original image, Saliency map,Groundtruth labeled by people An example of saliency concentrating on detecting dominate objects

13. Context-Aware In some cases, salient objects cannot summarize all the image contents. Other sources such as context should take part in.

14. Outline • Authors • Study Background • Abstract • Context-Aware Saliency Detection • Results • Conclusion • Appendix

15. Abstract • A new type of saliency aims at detecting the image regions that represent the scene • The Algorithm is based on four psychological principles • Two applications are introduced: retargeting and summarization

16. Outline • Authors • Study Background • Abstract • Context-Aware Saliency Detection • Results • Conclusion • Appendix

17. Four Principles(1/2) • Local low-level considerations, including factors such as contrast and color. • Areas that have distinctive colors or patterns should obtain high saliency. • Conversely, homogeneous or blurred areas should obtain low saliency • values • 2. Global considerations, which suppress frequently-occurring features, while maintaining features that deviate from the norm. • Frequently-occurring features should be suppressed

18. Four Principles(2/2) • Visual organization rules, which state that visual forms may possess one or several centers of gravity about which the form is organized. • The salient pixels should be grouped together, and not spread all over the image • High-level factors, such as human faces. • The semantic items should have higher saliency

19. Local-global Single-scale Saliency dcolor(pi,pj) is the Euclidean distance between the vectorized patches piand pj in CIE L*a*b color space.dposition(pi,pj) is the Euclidean distance between the positions of patches pi and pj . Only considering the K most simi- lar patches in the local measurement. The single-scale saliency value of patch i at scale r is defined as the equation left.

20. Multi-scale Saliency Enhancement Background patches are likely to have similar patches at multiple scales, Searching K most similar patches in the local measurement in scale R1 = {r,0.5r,0.25r} Representing each pixel by the set of multi-scale image patches centered at it. The saliency at pixel i is taken as the mean of its saliency at different scales

21. Including the Immediate Context Areas that are close to the foci of attention should be explored significantly more than far away regions. • Foci( attented pixels) • The pixels whose saliency values are higher than threshold will be seen as the foci. • Weighed saliency dfoci(i) is the Euclidean positional distance between pixel i and the closest focus of attention pixel, normalized to the range [0;1].

22. High-level Factors Facemap is the output of a face detector. Viola, P. and M. Jones. Rapid object detection using a boosted cascade of simple features. 2001: Citeseer.

23. Steps

24. Outline • Authors • Study Background • Abstract • Context-Aware Saliency Detection • Results • Conclusion • Appendix

25. Saliency Map Comparision of saliency result of 3 methods. [24] D.Walther and C. Koch. Modeling attention to salien objects. Neural Networks, 19(9):1395–1407, 2006. [7] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR, pages 1–8, 2007.

26. Saliency Map Comparing the saliency map in the paper with [13]. Top: Input images. Middle: the bounding boxes obtained by [13] capture a single main object. Bottom: the saliency map convey the story [13] T. Liu, J. Sun, N. Zheng, X. Tang, and H. Shum. Learning to Detect A Salient Object. In CVPR, 2007.

27. Image Retargeting Image retargeting aims at resizing an image by expanding or shrinking the non-informative regions. [19] M. Rubinstein, A. Shamir, and S. Avidan. Improved seam carving for video retargeting. ACM Trans. on Graphics, 27(3), 2008.

28. Summarization Extracting ROI by considering both the saliency and the image-edge information Computing the saliency maps Assembling these ROIs

29. Summarization Saliency maps The collage summarization Summarization of a trip to LA using 14 images.

31. Outline • Authors • Study Background • Abstract • Context-Aware Saliency Detection • Results • Conclusion • Appendix

32. Conclusion • Proposing a new type of saliency aims at detecting the important parts of the scene • Four principles were introduced • Local low-level considerations • Global considerations • Visual organizational rules • High-level factors • Approved by two applications : retargeting and summarization

33. Conclusion

34. Appendix Mentioned saliency detection methods

35. Local: Itti & Koch 1998

36. Global: Spectral Residual SR(Saliency) General Shape Log Spectrum Curve

37. Global: Spectral Residual A(f) is the amplitude of Fourier Transform and P(f) is the phase spectral of image.g(x) is a gaussian filter. are Fourier Transform and Inverse Fourier Transform respectively

38. Global : Spectral Residual

39. Local + Global: CRF Salient Object Features Multi-scale contrast Multi-scale contrast feature is defined as a linear combination of contrast in the Gaussian image pyramid. Il is the lth-level image in the pyramid and N(x) is the neighborhoods of the pixel x. fc(x,I) is normalized to a fix range [0,1].

40. Local + Global: CRF Salient Object Features Multi-scale contrast From left to right: the original image, contrast map at multiple scales, the feature map. Multi-scale contrast highlights the high contrast boundaries by giving low scores to the homogenous regions inside the salient object.

41. Local + Global: CRF Salient Object Features Center-surround histogram R(x’) Rs(x’)

42. Local + Global: CRF Salient Object Features Center-surround histogram Top: input images. Bottom: center-surround feature maps. The salient objects are well located by the center-surround histogram feature.

43. Local + Global: CRF Salient Object Features Color spatial distribution The wider a color is distributed in the image, the less possible a salient object contains this color. Compute the spatial variance of the color

44. Local + Global: CRF Salient Object Features Color spatial distribution All color in the image are represented by GMM Each pixel is assigned to a color component with the probability.

45. Local + Global: CRF Salient Object Features Color spatial distribution The horizontal variance of the spatial position for each color component. xhis the x- coordinate of pixel x. the vertical variance is similar defined.

46. Local + Global: CRF Salient Object Features Color spatial distribution Color spatial distribution, the left column are input images, and the right are feature map.

47. Local + Global: CRF Formulation The probability of the label A = {ax}(whether this pixel is salient) given the image I is directly modeled as a conditional distribution: Where Fk(ax,I) is the kth saliency feature and S(ax, ax’ ,I) is pairwise feature.λk is the weight of the kth feature, and x,x’ are two adjacent pixels.

48. Local + Global: CRF Formulation Each feature provide a feature map fk(x,I)~[0,1] for every pixel. The saliency feature is defined as the left equation. Where d x,x’ is L2 norm of the color difference .β is a robust parameter weighting the color contrast and can be set β = (2<||Ix,I x’||>) -1 .

49. Local + Global: CRF Results Example of salient features. From left to right: Input image, multi-scale contrast, center-surround histogram, color spatial distribution, and binary salient mask by CRF

50. Local + Global: CRF Results Salient Objects