Texture Optimization for Example-based Synthesis

1. Texture Optimization for Example-based Synthesis Vivek KwatraIrfan EssaAaron BobickNipun Kwatra GVU Center/College of Computing, Georgia Tech

2. Objects vs. Textures • Objects • Computer Graphics Primitives • Described by structure

3. Objects vs. Textures • Textures • Characterized by appearance • Elements arranged quasi-randomly

4. Dynamic Textures • Appearance similar across evolution

5. Natural Scenes contain Texture Vegetation Sand Water Clouds Rocks Snow

6. Goal: Image Extension

7. Goal: Video Extension Output: 30% Larger SpatiallyCan play it infinitely

8. Motivation + = InputTexture InputFlow Field OutputSequence

9. Contributions I • Global Optimization for texture synthesis • Global texture quality metric • Iterative multi-scale optimization

10. + = InputTexture InputFlow Field OutputSequence Contributions II • Flow-guided texture animation • Flow incorporated into optimization • Applications: fluid rendering, flow visualization

11. Flow-guided Texture Animation • Flow consistency • Perceived motion similar to flow • Texture similarity • Shape, size, orientation of texture elements similar to source Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

12. Source Texture Texture Similarity Naive Approach • Ignore Texture Similarity • Warp each frame via flow field Source Texture Warp TargetFrames X0 X1 Xn Texture Similarity FlowConsistency Flowing Target Target Flow

13. Naive Approach Texture structure not maintained Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

14. Warp X0 X1 Xn Instead of… Warp TargetFrames

15. Warp X0 X1 Xn Instead of… Warp TargetFrames

16. SynthesizedFrame X– X Warp Correct W WarpedFrame Our Approach Warp + Correct TargetFrames

17. “Correct” Operation Link to original goals TargetFrames X– X Correct Warp W

18. “Correct” Operation Link to original goals TargetFrames X– X Correct Flow Consistency Warp W

19. “Correct” Operation Link to original goals Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

20. Optimization of Energy / Cost Energy = Flow Energy + Texture Energy Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

21. Flow Energy TargetFrames X– X Warp Flow Energy W

22. Texture Energy • Tricky! • Pixel-by-pixel comparison of source and target does not make sense • Compare texture “elements” • Local pixel neighborhoods • Want each target neighborhood to be similar to some source neighborhood

23. Texture Energy Z(source) X(target frame)

24. Energy for Single Neighborhood Z(source) p(pixel) X(target frame)

25. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

26. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

27. Energy for Single Neighborhood (nearest neighbor) Zp Z(source) Xp(neighborhood) X(target frame)

28. Energy for Single Neighborhood (nearest neighbor) Zp Z Xp(neighborhood) Texture Energy(single neighborhood) X

29. Texture Energy for Entire Image Z X

30. Texture Energy for Entire Image Z X =  individual neighborhood energy

31. Optimization • Optimize Total Energy of target frame: • Initialize: X W(target frame  warped frame) • Iteratively improve target frame

32. Optimization: Intuition Zp Texture Force Xp Source Target Frame Flow Force Wp Warped Frame

33. Optimization: Intuition Zp Xp Blend Source Target Frame Wp Warped Frame

34. Overlapping Neighborhoods Zp Xq Zq Blend Source Xp Target Frame Wp Warped Frame

35. We are not done yet! Nearest neighbor may change Zp Xp Source Target Frame Wp Warped Frame

36. Iterative Algorithm • Step 1 • Find Nearest Source Neighborhoods • Step 2 • Blend Source & Warped Neighborhoods • Repeat until convergence

37. Minimize Texture Energy Alone • Novel Static Texture Synthesis Algorithm Source Texture Similarity X– X Correct Flow Consistency W

38. Texture Energy Graph • Random Initialization • Multiple • Resolution Levels • Neighborhood Sizes • Progressively refined output

39. Results: Texture Synthesis

41. Results: Texture Synthesis Input Wei-Levoy[Wei’00] Image Quilting[Efros’01] Graph-cuts[Kwatra’03] Texture Optimization[Kwatra’05]

42. Results: Texture Synthesis Image Quilting[Efros’01] Graph-cuts[Kwatra’03] Texture Optimization[Kwatra’05]

43. Results: Flow-guided Synthesis

44. Results: Timing • Static texture synthesis • 5-10 iterations per resolution level • Flowing texture synthesis • 20-60 seconds per frame

45. + = InputTexture InputFlow Field OutputSequence Summary • Global Optimization for controllable texture synthesis • Explicit evaluation of texture quality • Flow-guided texture animation within optimization framework

46. Limitations and Future Work • Optimization finds local minima • Blurry / misaligned texture elements • Extensions • Controllable video textures • Other control criteria like shape, illumination

47. Thank You