Shift-Map Image Editing

1. Shift-Map Image Editing Yael Pritch, Eitam Kav-Venaki, Shmuel Peleg Computer Science and Engineering The Hebrew University of Jerusalem, Israel ICCV 2009

2. Outline • Introduction • Image Editing as Graph Labeling • Hierarchical Solution for Graph Labeling • Shift-Map Application • Concluding Remarks

3. Outline • Introduction • Image Editing as Graph Labeling • Hierarchical Solution for Graph Labeling • Shift-Map Application • Concluding Remarks

4. Introduction • Geometric image rearrangement is becoming more popular • Image resizing (a.k.a. retargeting) • Object rearrangement and removal • Early methods manipulation mostly crop and scale • For image resizing, examining image content and removing “less important” regions

5. Introduction • Seam carving [2, 13] • Continuous image warping [19, 16] • Shift-map editing • Avoids scaling and mostly remove or shift image regions

6. (a) Original image (b) Video-retargeting [19] (c) Optimized scale-and-stretch [16] (d) Improved Seam Carving[13] (e) Our shift-map editing

7. (a) Original image (b) Our shift-map editing (c) Video-retargeting [19] (d) Optimized scale-and-stretch [16] (e) Improved Seam Carving[13]

8. Outline • Introduction • Image Editing as Graph Labeling • Hierarchical Solution for Graph Labeling • Shift-Map Application • Concluding Remarks

9. Image Editing as Graph Labeling • Shift-map • The relative shift of every pixel in the output image from its source in an input image • Represents the selected label for each output pixel • Two terms are used in computing the optimal shift-map • Data term • Smoothness term

10. Image Editing as Graph Labeling • Input image I(x, y) • Output image R(u, v) • The relationship between input image and output image is defined by • Shift-map M(u, v) = ( , ) • R(u, v) = I(x + , y + ) • Each output pixel can be labeled by a shift( , )

11. Image Editing as Graph Labeling • The optimal shift-map Mminimizes the cost function : • : data term • : smoothness term • N : neighboring pixels • = 1

12. Single pixel data term • Pixel rearrangement • Pixel saliency and removal • S : saliency map, very high for pixels to be removed, very low for pixels not to be removed

13. Smoothness term for pixels pair • The smoothness term represents discontinuities added to the output image by discontinuities in the shift-map • Two neighboring location and in the output image R if • The smoothness term account color difference and gradient difference

14. Smoothness term for pixels pair • : four unit vectors - four spatial neighbors • Color differences are Euclidean distances in RGB • and are the magnitude of the image gradients at these locstion • = 2

15. Outline • Introduction • Image Editing as Graph Labeling • Hierarchical Solution for Graph Labeling • Shift-Map Application • Concluding Remarks

16. Hierarchical Solution for Graph Labeling • Finding the optimal graph labeling, the number of possible labels is the number of pixels in the input image • Use heuristic hierarchical approach reduces the memory and computational • First solved in a coarse resolution • Higher resolution level

17. Hierarchical Solution for Graph Labeling • Example : 4th pyramid level • The number of pixels and number of labels are reduce by a factor of 64 Coarse shift-map Coarse level 4x4 Coarse level 16x16 Nearest neighbor interpolation 32x32 64x64 Input image Output image

18. Hierarchical Solution for Graph Labeling • Use three to five pyramid levels • The coarsest level contains up to 100 x 100 pixels

19. Outline • Introduction • Image Editing as Graph Labeling • Hierarchical Solution for Graph Labeling • Shift-Map Application • Concluding Remarks

20. Shift-Map Application • Image retargeting • Image rearrangement • Inpainting • Image composition

21. Image retargeting • Label order constraint • The shift-map will retain the spatial order • In the case of reducing width and , , • In the case of increasing width and , ,

22. Image retargeting • Controlling object removal • It is possible to control the size and number of removed objects by performing several steps of resizing • Also possible to control object removal by marking objects as salient

23. The number of steps becomes the number of removed columns • Original image • (b) Resizing in single step • (c) Six smaller resizing steps • (d) Ten smaller resizing steps

24. Shift-map retargeting : (a) Original image (b)(c)(e) No saliency (d) Child was marked salient

25. Original image [13] [19] [16] shift-map

26. Image rearrangement • Moving an object to a new image location • Deleting part of the image • Specified in two parts using the data term • Force pixels to appear in a new location using Eq. 2 • Marks these pixels for removal from their original location using Eq. 3

27. Example – 1 : • move the person and a part of the temple to the right, and keep the tourists at their original location

28. Example – 2 : • Kid on the left should move to the center, baby should move to the left, kid on the right should remain in place

29. Inpainting • Unwanted pixels are given an infinitely high data term as described in Eq. 3 • Maps pixels inside the hole to other locations in the input image

30. Inpainting • A good complition with no user intervention

31. Image composition • In the shift-map framework the input can consist of either a single image, or of a set of images • , is the index of the input image • Tolerate misalignments between the input images

33. Outline • Introduction • Image Editing as Graph Labeling • Hierarchical Solution for Graph Labeling • Shift-Map Application • Concluding Remarks

34. Concluding Remarks • Shift-maps are proposed as a new framework to describe various geometric rearrangement problems • Images generated by the shift map are natural looking • Minimal and intuitive user interaction • Distortions that may be introduced by stitching are minimized • Large regions can be synthesized

35. Thank you!!!