Inverse Texture Synthesis

1. Inverse Texture Synthesis Li-Yi Wei1 Jianwei Han2 Kun Zhou1,2 Hujun Bao2 Baining Guo1 Harry Shum1 1Microsoft 2Zhejiang University

2. Example-based texture synthesis • For a small input texture • produce an arbitrarily large output with similar look • Why? may not possible to obtain large input texture synthesis input output

3. Inverse texture synthesis • From a large input texture • produce a small output that best summarizes input inverse texture synthesis output input

4. Why? • Textures are getting large • Advances in scanning technology • High dimensionality: time-varying, BRDF • Expensive to store, transmit, compute Yale University MSR Asia Columbia University

5. Overview inverse texture synthesis input (large) output (small) texturing (fast) texturing (slow) similar quality

6. Related work: image compression pixel-wise identical compress decompress inverse synth texture synth input perceptual similar

7. Related work: epitome • Epitome [Jojic et al. 2003] • Jigsaw [Kannan et al. 2007] • Major source of inspiration for us • For general images, not just textures • We provide better quality • Bidirectional similarity [Simakov et al. 2008] • Factoring repeated content [Wang et al. 2008]

8. Related work: manual crop stationary globally varying original manual crop our result

9. Globally-varying textures • Markov Random Field (MRF) textures • local & stationary • Globally-varying textures • local, but not necessarily stationary MRF globally varying

10. Globally varying texturesPrevious work MRF input → globally varying output texture-by-numbers in Image analogies [Hertzmann et al. 2001] progressively variant textures [Zhang et al. 2003] texture design and morphing [Matusik et al. 2005] Globally varying input appearance manifold [Wang et al. 2006] spatially & time varying BRDF [Gu et al. 2006] context-aware texture [Lu et al. 2007]

11. Globally varying texturesDefinition texture + control maps Examples of control maps user-specified colors [Hertzmann et al. 2001] spatially-varying parameters [Gu et al. 2006] weathering degree-map [Wang et al. 2006] context information [Lu et al. 2007] texture (paint crack) control map (paint thickness)

12. Globally varying textures Including time-varying textures as well Large data size! time-varying BRDF [Gu et al. 2006] 512 x 512 x 33, 288 MB context-aware texture [Lu et al. 2007] 1226 x 978 x 50, 35 MB

13. Inverse texture synthesis Compacting globally varying textures including both texture + control map inverse synthesis texture control texture control map input output compaction

14. Compaction as summary of original • Re-synthesis with user control map faster slower forward synthesis + compaction user control re-synthesis from compaction re-synthesis from original

15. inverse term (New!) forward term [Kwatra et al. 2005] Basic formulation • Inspired by texture optimization [Kwatra et al. 2005] xp Zp best match zq best match Z (output) xq X (input)

16. Energy plot energy original compaction size

17. Why both terms? inverse forward • inverse term preserves all input features • forward term avoids artifacts in compaction both both f-only missing feature i-only garbage both i-only discontinuity

18. Comparing with epitome [Jojic et al. 2003] • Similar to our method but only inverse term • blur, discontinuity epitome epitome our our original original

19. Comparing with epitome [Jojic et al. 2003]Re-synthesis epitome epitome our our original original

20. Solver • How to solve this? • Texture optimization [Kwatra et al. 2005] • Discrete solver [Han et al. 2006]

21. NO inverse term forward term [Kwatra et al. 2005] zq xq Optimization [Kwatra et al. 2005] • E-step • fix xq • argminz E(x,z) • least square • M-step • fix Z • argminxq |xq-zq|2 • search fix xq xq Zq Z argminxq |xq-zq|2 X

22. inverse term forward term [Kwatra et al. 2005] zp zq xp xq Our solver • E-step • fix xq • argminz E(x,z) • least square • M-step (forward) • fix Z • argminxq |xq-zq|2 • search xp xq Zq discrete solver [Han et al. 2006] • M-step (inverse) • fix xp • argminzp |xp-zp|2 • discrete solver Z argminxq |xq-zq|2 discrete solver X

23. Results

24. GPU synthesis – small texture betterExtension from [Lefebvre & Hoppe 2005] 3 fps, original 6 fps, compact cheese mold 1214 x 1212 1282 3.5 fps, original 7.0 fps, compact dirt 271x481 1282 original compaction

25. Limitation:Correlation between texture & control texture control original reconstruction compaction

26. Orientation field for anisotropic textures • Orientation field w as part of energy function • E(x, z) → E(x, z; w) • Good orientation field yields better solution comp. no w comp. with w original orientation field

27. Future work • Higher dimensional textures • e.g. video • General images, not just textures • Bidirectional similarity [Simakov et al. CVPR 2008] • Image compression

28. Acknowledgements • Yale graphics group • Columbia graphics group • Sylvain Lefebvre • Hughes Hoppe • Matusik et al. 2005 • Mayang.com • Jiaping Wang • Xin Tong • Jian Sun • Frank Yu • Bennett Wilburn • Eric Stollnitz • Dwight Daniels • Reviewers • Dinesh Manocha • Ming Lin • Chas Boyd • Brandon Lloyd • Avneesh Sud • Billy Chen

29. Thank You!