Context-Aware Saliency Detection

Context-Aware Saliency Detection StasGoferman LihiZelnik-Manor AyelletTal Technion

Outline • 1. Introduction • 2. Principles of context-aware saliency • 3. Detection of context-aware saliency • 4. Result • 5. Applications

Context-aware saliency • Aim at detecting the image regions that represent the scene. • This definition differs from previous definitions whose goal is to either identify fixation points or detect the dominant object

What most people think is important or salient

Four Principles(1/2) 1. Local low-level considerations, including factors such as contrastand color. • Areas that have distinctive colorsor patterns should obtain high saliency • Conversely, homogeneousorblurredareas should obtainlowsaliency values 2. Global considerations, which suppress frequently- occurring features, while maintaining features that deviate from the norm.

Four Principles(2/2) • Visual organization rules, which state that visual forms may possess one or several centers of gravityabout which the form is organized. • The salient pixels should be grouped together, and not spread all over the image • High-level factors, such as human faces. • Implemented as post-processing.

Four principles • 1. Local low-level (b) • 2. Global (c) • 3. Visual organization rules about (b) + (c) • 4. High-level factors (post-processing)

Outline • 1. Introduction • 2. Principles of context-aware saliency • 3. Detection of context-aware saliency 3.1 Local-global single-scale saliency 3.2 Multi-scale saliency enhancement 3.3 Including the immediate context 3.4 High-level factors • 4. Result • 5. Applications

Local-global single-scale saliency Dissimilarity measure between a pair of patches as: dcolor(pi,pj) is the Euclidean distance between the vectorized patches pi and pj in CIE L*a*b color space. dposition(pi,pj) is the Euclidean distance between the positions of patches pi and pj . Single-scale saliency value of pixel i at scale r is defined as: Only considering the K most similar patches in the local measurement.

Multi-scale saliency enhancement Background patches are likely to have similar patches at multiple scales, Searching K most similar patches in the local measurement in scale R1 = {r,0.5r,0.25r} Representing each pixel by the set of multi-scale image patches centered at it. The saliency at pixel i is taken as the mean of its saliency at different scales

Steps • The steps of our saliency estimation algorithm *Multiple scales  foreground *Few scales  background

Including the immediate context • 1: A pixel is considered attended if its saliency value exceeds a certain threshold ( Si > 0.8). • 2: The saliency of a pixel is redefined as Let dfoci(i) be the Euclidean positional distance between pixel i and the closest focus of attention pixel, normalized to the range [0,1]

High-level factors • Final , face detection or recognized objects face detection algorithm of [23], which generates 1 for face pixels and 0 otherwise.

Result(1)

Result(2) Comparing the saliency map in the paper with [13]. Top: Input images. Middle: the bounding boxes obtained by [13] capture a single main object. Bottom: the saliency map convey the story [13] T. Liu, J. Sun, N. Zheng, X. Tang, and H. Shum. Learning to Detect A Salient Object. In CVPR, 2007.

Outline • 1. Introduction • 2. Principles of context-aware saliency • 3. Detection of context-aware saliency • 4. Result • 5. Applications 5.1. Image retargeting 5.2. Summarization through collage creation

Seam Carving for Content-Aware Image Resizing

Image retargeting • Image retargeting aims at resizing an image by expanding or shrinking the non-informative regions.

Summarization 1.Computing the saliency maps 3.Assemble the non-rectangular ROIs, allowing slight overlaps 2.Extracting ROI by considering both the saliency and the image-edge information

Summarization through collage creation (b)The collage summarization

Context-Aware Saliency Detection