1 / 62

Our Setting: 2D Texture Synthesis

Fast and High Quality Overlap Repair for Patch-Based Texture Synthesis Andrew Nealen Marc Alexa Discrete Geometric Modeling Group (DGM) Technische Universität Darmstadt. n x m Input Texture. N x M Output Texture. Our Setting: 2D Texture Synthesis.

bishop
Download Presentation

Our Setting: 2D Texture Synthesis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Fast and High Quality Overlap Repair for Patch-Based Texture SynthesisAndrew NealenMarc AlexaDiscrete Geometric Modeling Group (DGM)Technische Universität Darmstadt

  2. nxm Input Texture NxM Output Texture Our Setting: 2D Texture Synthesis • The goal: Synthesize an output texture which isperceptually similarto the input texture. Also ensure that the result contains sufficient variation.

  3. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  4. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  5. A B Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  6. A B Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  7. A B Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  8. A B 2 _ Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  9. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  10. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  11. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  12. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  13. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  14. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

  15. Patch-Based Texture SynthesisHybrid Texture Synthesis (HTS) • Introduced at EGSR 2003 [Nealen and Alexa] • Adaptive Patch Sampling, like Hierarchical Pattern Mapping [Soler et. al 2002] • Per-Pixel Overlap Re-synthesis

  16. Input (n x m) Intermediate Result Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

  17. Input (n x m) Intermediate Result Hybrid Texture SynthesisMethod Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M)

  18. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod Goal: From nxm, synthesize NxM similar, but not identical

  19. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

  20. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

  21. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

  22. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

  23. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod Goal: From nxm, synthesize NxM similar, but not identical

  24. Hybrid Texture SynthesisGeneralization: Pro and Con • Pro: General Method for Overlap Repair • Complementary to other Methods, such as Minimum-Error-Boundary-Cut (MEBC) or Feathering, yet oftentimes produces better results • Generalizes to arbitrary patch shapes, i.e. is applicable to Graphcut Textures, Wang Tiles, etc. • Con: Computationally Expensive • Exhaustive search for each invalid pixel in the overlap, based on mostly irregular valid neighborhood • Has O(rN log N) complexity -> Doesn‘t scale well.

  25. Hybrid Texture SynthesisGeneralization: Pro and Con • Pro: General Method for Overlap Repair • Complementary to other Methods, such as Minimum-Error-Boundary-Cut (MEBC) or Feathering, yet oftentimes produces better results • Generalizes to arbitrary patch shapes, i.e. is applicable to Graphcut Textures, Wang Tiles, etc. • Con: Computationally Expensive • Exhaustive search for each invalid pixel in the overlap, based on mostly irregular valid neighborhood • Has O(rN log N) complexity -> Doesn‘t scale well.

  26. Hybrid Texture SynthesisGeneralization: Pro and Con • Pro: General Method for Overlap Repair • Complementary to other Methods, such as Minimum-Error-Boundary-Cut (MEBC) or Feathering, yet oftentimes produces better results • Generalizes to arbitrary patch shapes, i.e. is applicable to Graphcut Textures, Wang Tiles, etc. • Con: Computationally Expensive • Exhaustive search for each invalid pixel in the overlap, based on mostly irregular valid neighborhood • Has O(rN log N) complexity -> Doesn‘t scale well.

  27. Fast Overlap RepairBasic Idea • Inspiration • Ashikhmin: Synthesizing Natural Textures [2001] termed Coherence Search • Tong et. al‘s extension: k-Coherence Search [2002] • Basic Idea: Intelligently Reduce Search Space • Only search within a set of coherent pixels • Introduce Trade-off between quality and speed

  28. Input Texture Intermediate Result + Source Map Fast Overlap Repair Coherence Search • Applying Coherence Search • For each pixel in the output, store its location in the input in a source map (same size as the output texture)

  29. Source Map Lookup Input Texture Intermediate Result + Source Map Fast Overlap Repair Coherence Search • Applying Coherence Search • When searching for a new pixel, only consider input pixels which are coherent with neighboring output pixels

  30. Source Map Lookup Input Texture Intermediate Result + Source Map Consequence in this example: Only two possible candidates Fast Overlap Repair Coherence Search • Applying Coherence Search • When searching for a new pixel, only consider input pixels which are coherent with neighboring output pixels

  31. Example: 64x64 Texture Synthesized from four 32x32 Patches Coherence Exhaustive Fast Overlap Repair Coherence Search • Applying Coherence Search • Simply comparing to the coherent pixels results in seams similar to Image Quilting (MEBC)

  32. Example: 64x64 Texture Synthesized from four 32x32 Patches Coherence Exhaustive Fast Overlap Repair Coherence Search • Applying Coherence Search • Simply comparing to the coherent pixels results in seams similar to Image Quilting (MEBC)

  33. Example: 64x64 Texture Synthesized from four 32x32 Patches Coherence Exhaustive Fast Overlap Repair Coherence Search • Applying Coherence Search • Simply comparing to the coherent pixels results in seams similar to Image Quilting (MEBC)

  34. Source Map Lookup Input Texture Intermediate Result + Source Map Fast Overlap Repair k-Coherence Search • Better: Applying k-Coherence Search

  35. Source Map Lookup Input Texture Intermediate Result + Source Map Fast Overlap Repair k-Coherence Search • Better: Applying k-Coherence Search • Extend the set by the k-nearest neighbors (knn) of each coherent pixel (in feature space) and remove duplicates

  36. Fast Overlap Repair k-Coherence Search • Precomputation of knn Data Structure • Performed once for each nxm input texture and stored for repeated use • User defines size of box-shaped neighborhood np • For each of the nxm input pixels • Construct feature vector by ordered concatenation of the npx npRGB-triples in the box-shaped neighborhood • Dimension reduction (75-90%) by applying PCA • Compute indices of k-nearest neighbors to each pixel

  37. Fast Overlap Repair k-Coherence Search • Source Map Maintenance • Each valid pixel in the overlap region is a linear blend (feathering) of at least two original pixel values, i.e. from at least two different sources • To avoid the maintenance of multiple source maps, simply store the source of the pixel with greatest contribution in a singlesource map Blue: invalid overlap pixels

  38. k = 1 k = 4 k = 11 Exhaustive Results varying k

  39. k = 1 k = 4 k = 11 Exhaustive Results varying k

  40. k = 1 k = 4 k = 11 Exhaustive Results varying k

  41. k = 1 k = 4 k = 11 Exhaustive Results varying k

  42. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  43. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  44. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  45. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  46. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  47. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  48. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  49. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

  50. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

More Related