1 / 56

Overview of Texture Synthesis

Overview of Texture Synthesis. Ganesh Ramanarayanan Cornell Graphics Seminar. ???. small texture. big texture. The Problem. ???. small texture. big texture. A Simple Solution …. small texture. A Simple Solution … Tiling. ???. big texture. A Simple Solution … Tiling.

dixon
Download Presentation

Overview of Texture Synthesis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of Texture Synthesis Ganesh Ramanarayanan Cornell Graphics Seminar

  2. ??? small texture big texture The Problem

  3. ??? small texture big texture A Simple Solution …

  4. small texture A Simple Solution … Tiling ??? big texture

  5. A Simple Solution … Tiling small texture Tiling is obvious! big texture?

  6. Ideal Solution?

  7. The Problem … in Words • Given texture I, generate a texture J which • Looks like the same texture • Has no obvious copying or tiling from I • Difference between I and J should be the same as the way I “differs from itself” [DeBonet97] • Things to watch for: • ‘Looks the same’: what is the texture model? • ‘Obvious copying’: how is it avoided? • Underlined text: indicates algorithm parameter

  8. Classes of Algorithms • Multiresolution pyramids • [HeegerBergen95] • Pixel-by-pixel synthesis • [EfrosLeung99] • Multiresolution pixel-by-pixel • [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01] • Patch quilting • [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] • Geometric feature matching • [WuYu04], [Liu et.al. 04]

  9. Heeger Bergen 1995 • Seminal paper that introduced texture synthesis to the graphics community • Algorithm: • Initialize J to noise • Create multiresolution pyramids for I and J • Match the histograms of J’s pyramid levels with I’s pyramid levels • Loop until convergence • Can be generalized to 3D

  10. Heeger Bergen 1995 - Algorithm • Image pyramids • Gaussian • Laplacian • Steerable pyramids [SimoncelliFreeman95] • b): multiple scales of oriented filters • c): a sample image • d): results of filters in b) applied to c)

  11. I I J J Heeger Bergen 1995 - Results Successes Failures

  12. Heeger Bergen 1995 - Results

  13. Heeger Bergen 1995 - Verdict • Texture model: • Histograms of responses to various filters • Avoiding copying: • Inherent in algorithm • No user intervention required • Captures stochastic textures well • Does not capture structure • Lack of inter-scale constraints

  14. De Bonet 1997 • Propagate constraints downwards by matching statistics all the way up the pyramid • Feature vector:multiscale collection of filter responses for a given pixel • Algorithm: • Initialize J to empty image • Create multiresolution pyramids for I and J • For each pixel in level of J, randomly choose pixel from corresponding level of I that has similar feature vector

  15. De Bonet 1997 - Algorithm • 6 feature vectors shown • Notice how they share parent information

  16. De Bonet 1997 - Results

  17. De Bonet 1997 - Verdict • Texture model: • Feature vector containing multiscale responses to various filters • Avoiding copying: • Random choice of pixels with ‘close’ feature vectors, but copying still frequent on small scale • Individual per-filter thresholds are cumbersome • Feature vectors used in later synthesis work

  18. Efros Leung 1999 • Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood • Algorithm: • Initialize J to empty • Synthesize new pixels in a spiral pattern • Select new pixel from I based on conditional probabilities given the current neighborhood in J and similar neighborhoods in I

  19. Efros Leung 1999 - Algorithm I J ‘causal’ neighborhood

  20. Efros Leung 1999 - Results

  21. Efros Leung 1999 – Verdict • Texture model: • MRF • Avoiding copying: • MRF • Neighborhood size = largest feature size • Markov model is surprisingly good • “I spent an interesting evening recently with a grain of salt.” • Search is very slow with large neighborhoods

  22. Wei Levoy 2000 • Adds tree-structured vector quantization speedup and multiresolution matching • Algorithm: • Initialize J to noise • Synthesize new pixels in scanline order • Select new pixel from I that has the closest matching neighborhood to J • In multiresolution case, look instead at neighborhood of feature vectors

  23. Wei Levoy 2000 - Algorithm • Scanline synthesis with neighborhoods • a) input texture • b) algorithm start • c) midway point • d) ending

  24. Wei Levoy 2000 - Results

  25. Wei Levoy 2000 - Verdict • Texture model: • Multiscale filters and MRF • Avoiding copying: • Not explicitly, but in practice inherently avoided; TSVQ adds additional unpredictability • Results comparable to [EfrosLeung99] • TSVQ speedup comes at a price (quality loss)

  26. Ashikhmin 2001 • Adds more verbatim image copying and user-constrained synthesis • Algorithm: • New array J* stores location in I of J’s pixels • Initialize J to empty, J* to random • Synthesize new pixels in scanline order • Select new pixel by copying from patches of I, as determined by locations in J* • In user-constrained case, also include error comparison with user-specified target image T

  27. Ashikhmin 2001 - Algorithm • Candidates: • x=4, y=5 • x=18, y=16 • x=11, y=12 x=3 y=4 x=4 y=4 x=12 y=11 x=17 y=16 x=18 y=16 J*

  28. Ashikhmin 2001 - Results

  29. Ashikhmin 2001 - Verdict • Texture model: • MRF • Avoiding copying: • Actually, here it is encouraged on a small scale, but in practice it doesn’t occur on a large scale • Recognized that copying preserves fine detail • Features can get ‘cut off’ because of the abrupt jumping from patch to patch

  30. Hertzmann et. al. 2001 • Framework for solving image analogies: learning an image filter f by example • Given A, f(A), and B, generate f(B) • To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J • Algorithm: • Bring image histograms into correspondence • Combine [WeiLevoy00] and [Ashikhmin01] Each search method is weighted appropriately and then the best choice is used

  31. Hertzmann et. al. 2001 - Algorithm • A provides set of constraints • Histogram matching critical because of color differences between A and B sets of images

  32. I I J J Hertzmann et. al. 2001 - Results

  33. Hertzmann et. al. 2001 - Results

  34. Hertzmann et. al. 2001 - Verdict • Texture model: • Multiscale filters and MRF • Avoiding copying: • See [WeiLevoy00] and [Ashikhmin01] • Unique approach to image manipulation and texture synthesis • Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]

  35. Efros Freeman 2001 • Inspired by chaos mosaics [Xu et. al. 00] • Make an ‘image quilt’ by pasting together patches of the input and fixing seams • Algorithm: • Initialize J to empty • Copy new patches of certain size in scanline order, with fixed overlap width • Randomly select new patch that has error in overlap region less than given threshold • Copy pixels based on minimal error boundary cut

  36. B1 B1 B2 B2 Neighboring blocks constrained by overlap Minimal error boundary cut Efros Freeman 2001 - Algorithm Input texture block

  37. Efros Freeman 2001 - Results

  38. Efros Freeman 2001 - Verdict • Texture model: • ?? • Avoiding copying: • Randomized patch selection, but still noticeable • Patch size is a hard parameter to understand • Results are surprisingly good given algorithm • Multiscale goes on a brief hiatus

  39. Kwatra et. al. 2003 • Generalizes seam computation in overlap regions as a graph cut problem • Based on [Boykov et. al. 99] (with Ramin Zabih) • Algorithm: • Initialize J to empty • Copy pieces of I to J using a variety of methods • Formulate flow graph in overlap region based on errors and compute minimum cut • Copy sink-side pixels to J • Variety of strategies to further hide seams

  40. Kwatra et. al. 2003 - Algorithm (assume cut region is 3x3 for simplicity)

  41. Kwatra et. al. 2003 - Results

  42. Kwatra et. al. 2003 - Results

  43. Kwatra et. al. 2003 - Verdict • Texture model: • MRF • Avoiding copying: • Even with a multitude of patch selection methods, still noticeable when it happens repeatedly • Paper presents a bag of synthesis tricks without much intuition for when to use what • Graph cut formalization is useful and powerful

  44. Wu Yu 2004 • Synthesizes a feature image J* in parallel • Tries to match features in overlap region • Algorithm: • Compute feature image I* • Initialize J* to empty • Synthesize J and J* in parallel • When searching for new patch, add in weighted feature matching error • Prior to copying, warp selected patch to match features exactly

  45. Wu Yu 2004 - Algorithm • Match feature pixels based on • Tangent orientation • Distance • Bijectivity • Example: • L1out matches L1in • L2out matches L3in • Warping will move matching features closer to each other Feature correspondence

  46. Wu Yu 2004: Results

  47. Wu Yu 2004: Verdict • Texture model: • Feature lines and MRF • Avoiding copying: • No; in fact, mirroring is often very noticeable • Unclear how to combine this metric with others • Insight to use warping is valuable, although the next paper does a much better job with it

  48. Liu et. al. 2004 • Regular textures: can be tiled • Near-regular textures: statistical distortion of a regular texture • Algorithm (simplified): • Mark lattice points in I • Transform I into regular texture R • Express transformation as deformation texture I* • Synthesize larger deformation J* • Synthesize J by combining J* with R

  49. Liu et. al. 2004 - Algorithm I R Deformation Texture I*

  50. Liu et. al. 2004 - Results

More Related