2D Texture Synthesis

1. 2D Texture Synthesis Instructor: Yizhou Yu

2. Texture synthesis • Goal: increase texture resolution yet keep local texture variation

3. Synthesis by global statistics • Idea: Obtain statistics of input image, match with output image • Histograms: normalized graph of intensity frequencies

4. Synthesis by global statistics • Consider extreme case: • Histogram matching can generate two images:

5. Using higher-order global stats • Histogram matching does not take into account spatial information. • Need additional info: • For every pixel, examine local neighborhood • Gives local but not global features (e.g. – veins) • Use higher order stats: • Correlation • average of the product of the intensities = • tells how interdependent pixel values are

6. General procedure • 1.) Define error metric: • What do you want to match? • 2.) Match statistics • Histogram matching: • Get the density value of a pixel in the output image. Map this density to a pixel intensity in the input image  Overwrite the pixel in the output image with the mapped pixel intensity in the input image

7. Pyramid-based Texture Analysis/Synthesis • Paper by David J. Heeger and James R. Bergen from SIGGRAPH 1995 • http://www.cns.nyu.edu/~david/ftp/reprints/heeger-siggraph95.pdf

8. Pyramid-based Synthesis • Downsample image several times and keep track of only differences in a feature image • To reconstruct, upsample small images and add differences • Pixels in feature images are close to zero • Provides multiple scales with each feature image representing different feature sizes

9. Laplacian Pyramid • 1-D filter: (1/16) (1 4 6 4 1) • To generalize to 2D, use tensor product • Normalize by dividing by 256

10. Upsampling • To upsample, copy pixels to every other pixel in larger image • Fill the rest with zeroes, run low pass filter to fill in values

11. Oriented Filters • Gaussian filters are symmetric, thus edges and contours are not detected • Use multiple oriented filters to catch non-symmetric features Left top series: Oriented filters Right image: Texture Left bottom series: Filtered textures

12. Pyramids with Oriented Filters • Each oriented filter creates its own feature image • Thus, for each downsampled image, keep one image for each oriented filter • Each oriented filter captures one orientation of lines

13. Matching Image Features • Input parameters: • noise: initial noisy texture • texture: texture to be matched • Output texture is stored in noise

14. Matching Image Features • Helper functions: • MatchHistogram(noise, texture) • Matches histogram, using method described last time • MakePyramid(texture) • Create pyramid images (base and feature images) • CollapsePyramid(pyramid) • Constructs high resolution image from base and feature images

15. Matching Image Features MatchTexture(noise, texture) { MatchHistogram(noise, texture) // first-order matching analysis_pyr = MakePyramid(texture) // create pyramid from input texture for several iterations synthesis_pyr = MakePyramid(noise) // create pyramid from noise for each feature image, fi of analysis_pyr for each feature image, fj of synthesis_pyr of same orientation MatchHistogram(fi, fj) end for end for noise = CollapsePyramid(synthesis_pyr) MatchHistogram(noise, texture) end for }

16. Matching Image Features • MatchTexture matches histograms of feature images, not pixel values, thus providing a much better matching • Feature images already consider local neighborhoods, resulting in better approximations • Good for randomized textures • Textures with large scale features or thin and long features not matched well • e.g. stripes in wood, lines in coral

17. Texture Synthesis Using Local Neighborhoods • Main goal is to keep local spatial coherence, but not global stats. • Randomized method • Pick pixel and copy it along with its neighborhood to random parts in synthesized image • If two neighborhoods overlap, just blend • This can result in features getting cut off if larger than local neighborhood

18. Neighborhood-Based Texture Synthesis • Patch-Based • Patch-based sampling achieves real-time speed. [Liang et. al. 2001] • Image quilting: high-quality results and simple implementation [Efros & Freeman 2001] • Graph cut provides a powerful and refinable scheme. [Kwantra et. al. 2003] • Pixel-Based • [Efros & Leung 99], [Wei & Levoy 2000], [Ashikhmin 2001], [Hertzmann et. al. 2001], [Zhang et. al. 2003]

19. Pixel-wise Synthesis • Grow pixel by pixel • Start from an existing patch of the texture (as opposed to noise texture like Pyramid-based Synthesis) • Look for regions in input texture most similar to current region in new texture • Copy pixels next to best-match region to expand new texture

20. Following Raster Order • Blue region: already set by algorithm • Green region: compare this region to input image • Yellow region: closest match • Red region: replace these pixels with magenta region to maintain local integrity

21. Hierarchical Synthesis • Build a multi-resolution pyramid for the example texture • Generate a synthesized pyramid for the output texture • At each level, follow the raster order

22. Randomizing Synthesis • Instead of picking pixels from closest matching region, use threshold to pick a few candidate matches • Randomly pick one of the candidates • Refer to “Texture Synthesis by Non-parametric Sampling” • http://www.cs.berkeley.edu/~efros/research/synthesis.html

23. Results

24. Patch-Based Synthesis • Search in a sample texture for neighborhoods most similar to a context region • Merge a patch with the partially synthesized output texture

25. The Seam Problem • Feature discontinuities may appear in patch-based synthesis.

26. The Second Reason • Didn’t find the smoothest transition between the incoming patch and context region. • Solutions: use dynamic programming [Efros and Freeman 2001] or graph cut [Kwatra et al. 2003] to find an optimal cut. Original Warped

27. The First Reason • Rigid template matching (SSD) often employed in the first step does NOT consider geometric similarity Original Warped

28. Feature Map Guided Texture Synthesis • Basic Steps [Wu and Yu 2004] • Maintain an input and output feature map • Match and align curvilinear features • Integrate feature maps into template-based texture synthesis

29. Comparisons Sample Feature Map Graphcut Quilting Texton Mask

30. Acceleration Schemes • Fourier Transform • Acceleration techniques for nearest-neighbor search • Tree-structured vector quantization • Kd-trees • Minimizing the candidate set • Coherent synthesis, [Ashikhmin 2001] • Precomputing candidate set • K-coherent search, [Tong et al. 2002] • Jump Map, [Zelinka and Garland 2002] • GPU with parallel synthesis • [Lefebvre and Hoppe 2005] Original Warped