An Introduction to Texture Synthesis

1. An Introduction to Texture Synthesis 2008/05/29

2. Desirable Properties Result looks like the input Efficient General Easy to use Extensible

3. The Challenge Need to model the whole spectrum: from repeated to stochastic texture repeated stochastic Both?

4. [Shannon,’48] proposed a way to generate English-looking text using N-grams: • Assume a generalized Markov model • Use a large text to compute prob. distributions of each letter given N-1 previous letters • Starting from a seed repeatedly sample this Markov chain to generate new letters • Also works for whole words

5. p non-parametric sampling Assuming Markov property, compute P(p|N(p)) • Building explicit probability tables infeasible • Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p To synthesize p, just pick one match at random Input image Synthesizing a pixel

6. Basic Algorithm Exhaustively search neighborhoods

7. Neighborhood Neighborhood size determines the quality & cost 33 55 77 739 s 423 s 528 s 1111 4141 99 24350 s 1020 s 1445 s

8. Efros & Leung ’99 The algorithm • Very simple • Surprisingly good results • Synthesis is easier than analysis! • …but very slow Optimizations and Improvements • Multi-resolution Pyramid • Jump Map • Chaos Mosaic

9. Multi-resolution Pyramid High resolution Low resolution

10. Multi-resolution Algorithm

11. Multi-resolution Pyramid Better image quality & faster computation 1 level 55 1 level 1111 3 levels 55

12. What is a Jump Map? Same size as input

13. What is a Jump Map? Same size as input Set of jumps per pixel

14. What is a Jump Map? 0.4 0.2 0.3 • Same size as input • Set of jumps per pixel • Jumps are weighted according to similarity • Need not sum to 1

15. Synthesis with Jump Maps Input Output

16. Chaos Mosaic [Xu, Guo & Shum, ‘00] Process: 1) tile input image; 2) pick random blocks and place them in random locations 3) Smooth edges input idea result Used in Lapped Textures [Praun et.al,’00]

17. Chaos Mosaic [Xu, Guo & Shum, ‘00] Of course, doesn’t work for structured textures input result

18. Image Quilting Idea: • let’s combine random block placement of Chaos Mosaic with spatial constraints of Efros & Leung

19. Efros & Leung ’99 extended B p Synthesizing a block • Idea: unit of synthesis = block • Exactly the same but now we want P(B|N(B)) • Much faster: synthesize all pixels in a block at once • Not the same as multi-scale! non-parametric sampling Observation: neighbor pixels are highly correlated Input image

20. B1 B1 B2 B2 Neighboring blocks constrained by overlap Minimal error boundary cut block Input texture B1 B2 Random placement of blocks

21. Minimal error boundary 2 _ = overlap error min. error boundary overlapping blocks vertical boundary

22. Algorithm • Pick size of block and size of overlap • Synthesize blocks in raster order • Search input texture for block that satisfies overlap constraints (above and left) • Easy to optimize using NN search • Paste new block into resulting texture • use dynamic programming to compute minimal error boundary cut

27. Failures (Chernobyl Harvest)

28. ? + = Image Quiltingfor Texture Transfer

29. Texture Transfer Take the texture from one object and “paint” it onto another object • This requires separating texture and shape • That’s HARD, but we can cheat • Assume we can capture shape by boundary and rough shading Then, just add another constraint when sampling: similarity to underlying image at that spot

30. parmesan + = rice + =

31. + =

32. Source correspondence image Target correspondence image Source texture Target image

33. + =

34. Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Image Quilting

35. Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Image Quilting

36. Homage to Shannon! Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Image Quilting