Generating Classic Mosaics with Graph Cuts

1. Generating Classic Mosaics with Graph Cuts Y. Liu, O. Vekslerand O. Juan University of Western Ontario, Canada Ecole Centrale de Paris, France

2. Different Types of Mosaics Jesus at Basilica of Sant' Apollinare Nuovo A. Hausner [SIGGRAPH 01] K. Smith et. al [SIGGRAPH 05] J. Orchard et. al [NPAR 08]

3. Classic Mosaic: Main constraints • Tile orientation should emphasize important scene edges • Tiles should be packed tightly while preserving their completeness Good Tiling Bad Tiling

4. Previous Work • User Interaction • Hausner [SIGGRAPH2001] • Elber and Wolberg[Visual Computer 2003] • Explicit Edge detection • Di Blasi et. al [Visual Computer 2005] • Battiato et. al [WSCG 2006] • Implicit Methods • Battiato et. al [EG 2008] • Liu et. al [EMMCVPR 2007]

5. Previous Work Cont. • Requires user interaction • Discontinuities in tile orientation filed • No control over tile overlap • Local optimization for tile orientation • Heuristics for tile packing Battiato et. Al [EG 2008] Hausner [SIGGRAPH 2001]

6. Our Method: Overview • We address the problem in a global optimization framework • pixel labelling problem • Formulate energy which encodes constraints for a good mosaic • Encourage tiles to align to image edges • Smoothly varying tile orientations • Tight tile packing • Overlapping tiles prohibited Low Energy High Energy

7. Mosaic as Pixel Labelling • Each pixel p is assigned a pairwise label (vp , ϕp) • ϕp is tile orientation • if tile centered at p is visible, orientation is ϕp • vpis visibility • 1 means tile is present • 0 means tile is hidden 0 1 1 ϕp 0

8. Energy Function encourages smooth tile orientations and prohibits tile overlap encourages tiles to align to image edges encourages dense packing high cost high cost high cost high cost low cost low cost low cost

9. Optimization with Graph cut • NP-hard to optimize all variables simultaneously • We use step-wise optimization • First, ignore the visibility labels and optimize the orientation variables • Then, fix the orientation labels and optimize the visibility variables

10. Optimizing Orientation Variables • Assign an orientation label ϕp to each pixel • don’t know where the tiles will be placed, so optimize for all possible locations • approximately optimized by alpha-expansion [Boykov et.al. PAMI01]

11. Optimizing the visibility variables • Heuristically build many mosaic layers according to the orientation labels • Each layer consists of non-overlapping tiles • Each layer is as dense as possible

12. Stitching Layers 1 1 0 0 1 • Maintain current solution • Randomly pick mosaic layer • Find the smallest energy stitching 1 1 1 0 1 0 1 1 0 • formulated as binary labelling problem • energy roughly corresponds to the area of the gap space, therefore encouraging reduction in gaps • tiles will not overlap since energy would be infinite • Submodular energy, optimized with a graph cut [Kolmogorov et. al. PAMI2004]

13. Layer Stitching Illustration • Tiles may cross the image edges and create a blurry effect Current Solution Random Layer

14. Layer Stitching illustration • Tiles overlapping intensity edges are removed Bad stitching Good stitching

15. Overview of Layer Stitching • Start from a random layer • In each iteration, stitch the current solution and a candidate mosaic layer together • Repeat until the result is visually pleasing

16. Experimental Results Our Approach Battiato et. Al [EG 2008]

17. Experimental Results Our Approach Battiato et. Al [EG 2008]

18. Experimental Results

19. Experimental Results

20. Mosaic with Different Tile Size Input Image 10 by 10 20 by 20

21. Number of Iterations in Stitching Starting Layer 27 iteration 156 iteration

22. Edge Avoidance Small Weight Large Weight

23. Conclusion and future work • We address classic mosaic problem in an energy optimization framework • Our energy function encodes desired mosaic properties • Tiles align to image edges, minimize gap space, prohibit tile overlap • We eliminate monotonous user interaction of drawing feature edges • Take advantage of powerful discrete optimization methods based on graph cuts • Results are visually pleasing • Extensions: • tiles of different shapes and sizes • video mosaics