Seam Carving for Content-aware Image Resizing

Seam Carving for Content-aware Image Resizing 資訊碩一 10077034 蔡勇儀 Date：2012/01/03 @LAB 603

Outline • Introduction • Basic Theory • Application & Implementation • Aspect Ratio Change • Retargeting with Optimal Seams-Order • Enlarging • Content Ampliﬁcation • Seam Carving in the gradient domain • Object Removal • Multi-size Images • Limitation • Conclusions and Future Work

Introduction(1/2) • Muti-Media and Embedding System(e.g.Cell Phone) grow fast，Resize or Multi-Size scalingare more important than past. • Standard image scaling is not sufﬁcient since it is oblivious to the image content and typically can be applied only uniformly. • For improve the problem, many researcher prove some good idea.

Introduction(2/2) • The following is main methods for scaling： • Corp (figure2(b)) • Column orRow removal (figure2(c)) • Pixel energy removal (figure2(e)) • Optimal Pixel energy removal (figure2(f)) • Object detection • Seam Craving (figure2(d)) • We can found the seam method have better result!

Basic Theory • Step1 – Find seam • Find a path which have the minimum energy sum from image top to bottom. • Step2 – Remove the Min. seam • When found all seam, select the Min. seam remove. • Step3 – Repeat above step until get the demand size

Basic Theory – Step1 • Give an energy function • Define Seam • Define the Seam Cost

Basic Theory – Step2 • Find the minimum seam • Remove S* form image and lnsteadof neighbors

Basic Theory – Step3 • Repeat above step until get the demand size

Basic Theory – Energy Function • What’s energy function is the best? • e1 • Entropy • 9*9 Windows add to e1 • Segmentation ( add to e1) • Histogram of Gradients • 11*11cell around a pixel, 8-bins

Basic Theory – Energy Function • They all accommodate a similar range for resizing. • We found either e1 or eHoG to work quite well.

Aspect Ratio Change • Only one axis adjust • A picture size n*m  n*m’ or n’*m • n >= n’ m >= m’ • Remove n-n’ or m-m’ seams • Enlarge at other page

Retarget with Optimal Seam-Order(1/3) • What’s the optimal order for remove seams? Column or Row or Other? How could decide? • Using dynamic programming • where k = r+c, c = (m−m’), r = (n−n’) • αi is used as a parameter • that determine if at step i we remove a horizontal or vertical seam:

Retarget with Optimal Seam-Order(2/3) • Define transport map T • T(r,c)=min(T(r-1,c)+E(sy(In-r+1*m-c)), T(r,c-1)+E(sx(In-r*m-c+1)) ) • where In-r*m-c • denotes an image of size (n−r)×(m−c), • E(sx(I)) and E(sy(I)) • are the cost of the respective seam removal operation. • Build the 1 bit map for record the direction

Retarget with Optimal Seam-Order(3/3)

Enlarge(1/3) • When m’ > m or n’ > n, we should insert seams to the picture. • Find the smallest energy seam for copy and insert, repeat until equal the demand scale. • But…

Enlarge(2/3) • Every time found the same seam, so we should decide all seams which need copy at first. • If m’ > m then we need insert (m’-m) seams. Find them and copy it for insertation.

Enlarge(3/3) The origin picture Scalar Seam

Content Amplification • Using same scalar enlarge then use seams-carving for recover to the origin size.

Seam Carving in the gradient domain • If energy funciton use the gradient, then color show at remove place will be more nature after seam carving.

Object remove • User mark the part which want to remove. • Decrease the energy on the part is removed. • Insert seams for keeping origin size.

Multi-size Images(1/3) • User want find the optimal picture scalar for their demand, so we need the real time opreation. • But the picture’s size 400*500 to 100*100 in about 2.2 seconds, it is too long to real time. How could do for real time?

Multi-size Images(2/3) • Make the index map for seams before user operation. • Build the horizontal & vertical index map (H&V) • But there will a big problem for operation that is H & V will be collided. • The sample solution is decide one just do one direction and then other direction need degenerate the index and redo the select seams operation

Multi-size Images(3/3)

Limitation(1/2) • this method • does not work automatically on all images. • can be corrected by adding higher level cues, either manual or automatic. Figure 14, Figure 15 • Other times, • not even high level information can solve the problem. • two major factors that limit this seam carving approach. • The first • is the amount of content in an image. • If the image is too condensed, • it does not contain ‘less important’ areas, • then any type of content-aware resizing strategy will not succeed. • The second type of limitation • is the layout of the image content. • In certain types of images, albeit not being condensed,the content is laid out in a manner that prevents the seams to bypass important

Limitation(2/2)

Conclusions and Future Work • to extend this approach to other domains, • the first of which would be resizing of video. • Since there are cases • when scaling can achieve better results for resizing, • would like to investigate the possibility • to combine the two approaches, • Specifically to define more robust multi-size images. • would also like to find a better way • to combine horizontal and vertical seams in multi-size images.

Thank for your listening • Q&A

Seam Carving for Content-aware Image Resizing