Improved Seam Carving for Video Retargeting

1. Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume 27 Issue 3, August 2008 Presenter: Hon-Hang Chang Date: 2011/ 09/ 30

2. Outline Introduction Preliminaries Seam Carving using Graph Cut Forward Energy Results Conclusion

3. Because of human perception, the temporal may even be more disturbing than spatial invideo, as the human eye is highly sensitive to movement. 1 D path on 2D image (Seam) • 2D manifolds Introduction To treat video as 3D cube 2D manifolds in a 3D volume • In this paper the author define a new formulation of seam carving using graph cuts.

4. A whole seam A seam is a monotonic and connected path of pixels going from the topof the image to the bottom Introduction Or from left to right • The image size is reduced by one either in the horizontal or the vertical dimension. • video should support retargeting capabilities as it is displayed on TVs, computers, cellular phones.

5. Temporal term • SpatiotemporalL1-norm Spatial term Preliminaries Among this, {It} t=1~N Where, α ∈ [0, 1] Extend the spatial L1-norm to a spatiotemporal L1-norm: • Motion artifacts are more noticeable • Taking α= 0.3

6. Graph cut Seam Carving using Graph Cuts S (Source) arc1 T (Sink) arc2 Sink Source arc3 arc4

7. Optimal seam in graph cut • An optimal seam must be: • 1.Valid • - A simple cut cannot define a seam carving • 2. Monotonic • - Including only one pixel in each row (or column) • 3. Connected • -The seam must be connected Seam Carving using Graph Cuts • The proposed formulation extend seam carving to video and define a monotonic and connected 2D manifold seam inside the video cube.

8. Graph cut for image(Non-monotonic) • Every internal node P is connected to its four neighbors Nbr(Pi,j) = {Pi−1,j , Pi+1,j , Pi,j−1, Pi,j+1}. Seam Carving using Graph Cuts ∂y(i, j) = |I(i+1, j ) − I(i, j)| Pi-1,j Vertical direction Pi,j-1 Pi,j+1 Pi,j Pi+1,j Neighbors pixels ∂x(i, j) = |I(i, j + 1) − I(i, j)| Horizontal direction

9. Graph cut for image(Unconnected) Seam Carving using Graph Cuts • Prove_1:

10. Graph cut for image(Original) E1(i, j) =∂x(i, j) + ∂y(i, j) Seam Carving using Graph Cuts • Prove_2:

11. Graph cut for image • We can use any energy function defined on the pixels as the weight of the forward horizontal arcs . Seam Carving using Graph Cuts • Achieve the same results as the original dynamic programming based seam carving

12. Graph cut for video Seam Carving using Graph Cuts • The extension to video is straightforward.

13. Graph cut for video • Computation time is quadratic in the number of voxels. O(mn2) • - n  #of nodes ; m#of arcs Seam Carving using Graph Cuts • Performance issues are encountered already for high resolution images • The graph cut approach to seam carving allows us to extend the benefits of content-aware resizing to video. Still, the method is not perfect and no single energy function was shown to perform properly in all cases. Therefore, they introduce a new energy function that better protects media content, and improves video results.

14. Remove the seam with the least amount of energy from the image Forward Energy • The inserted energy is due to new edges created by previously non adjacent pixels that become neighbors once the seam is removed • To measure the real change in energy after a removal of a seam They measure the difference in the energy of the image after the removal (It=i+1) and the energy of only those parts that were not removed in the previous image

15. Forward Energy

16. Forward Energy in Graph Cut Forward Energy +LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1) difference between the Left and Right neighbors +LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j) difference between the Left and Up neighbors -LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j) difference between the Left and Up neighbors with respect to the end point of the arrow

17. Forward Energy in Graph Cut Forward Energy +LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1) difference between the Left and Right neighbors +LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j) difference between the Left and Up neighbors -LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j) difference between the Left and Up neighbors with respect to the end point of the arrow

18. Results

19. Results

20. Conclusion • Video retargeting is achieved using graph cuts and we have shown a construction that is consistent with the dynamic programming approach.