Lazy Solid Texture Syntehsis

1. Lazy Solid Texture Syntehsis Eurographics Symposium on Rendering 2008 Yue Dong, Sylvain Lefebvre, Xin Tong, George Drettakis

2. Abstract • We introduce a new algorithm with the unique ability to restrict synthesis to a subset of the voxels, while enforcing spatial determinism • Only a thick layer around the surface needs to be synthesized • Synthesize a volume from a set of pre-computed 3D candidates • Carefully select in a pre-process only those candidates forming consistent triples

3. Abstract • Runs efficiently on the GPU • Generates high resolution solid textures on surfaces within seconds • Memory usage and synthesis time only depend on the output textured surface area • Our method rapidly synthesizes new textures for the surfaces appearing when interactively breaking or cutting objects

4. Introduction • Solid textures define the texture content directly in 3D • Removes the need of a planar parameterization • Unique feeling that the object has been carved out of a block of matter

5. Previous Work • Low memory usage • Limited range of materials • Implicit Volume • Color = f(x, y, z) • Procedural texturing • Texturing and Modeling: A Procedural Approach • EBERT D., MUSGRAVE K., PEACHEY D., PERLIN K., WORLEY • Academic Press, 1994 • Spectral analysis • Spectral analysis for automatic 3d texture generation • GHAZANFARPOUR D., DISCHLER J.-M. • Computers & Graphics, 1995 • Generation of 3d texture using multiple 2d models analysis • GHAZANFARPOUR D., DISCHLER J.-M. • Computers & Graphics,1996

6. Previous Work • Good Quality • Can synthesis various materials • Take long time to compute • Explicit Volume • Color = g[x, y, z] • Histogram matching • Pyramid-Based texture analysis/synthesis • HEEGER D. J., BERGEN J. R. • SIGGRAPH, 1995 • Stereological technique • Stereological techniques for solid textures • JAGNOW R., DORSEY J., RUSHMEIER H. • SIGGRAPH, 2004 • Neighborhood matching • Texture synthesis by fixed neighborhood searching • WEI L.-Y. • PhD thesis, 2002, Stanford University • Aura 3d textures • QIN X., YANG Y.-H. • IEEE Transactions on Visualization and Computer Graphics, 2007 • Solid texture synthesis from 2d exemplars • KOPF J., FU C.-W., COHEN-OR D., DEUSSEN O., • LISCHINSKI D., WONG T.-T. • SIGGRAPH, 2007

7. Process Overview • Pre-computation • 3D candidates from 2D exemplars • Multi-resolution pyramid synthesis • Upsample • Jitter • Correction

8. Terminology • Pixel : 2D / Voxel : 3D • Triple : a set of three 2D coordinates • Crossbar : a set of pixels which are crossing in three neighborhoods of size N (N = 5)

9. 3D candidates selection • We select candidate triples following two important properties • A good triple must have matching colors along the crossbar of the three neighborhood • To provide color consistency • A good triple must have a good coherence across all three exemplars • Which is likely to form coherent patches with other neighboring candidates

10. Color consistency • A suitable candidate should be consistent across the crossbar • Minimize the color difference of the crossbar • Compute L2 color difference between each pairs • The sum of difference for the three pairs defines a crossbar error CB

11. Color consistency • In each pixel of each exemplar • Form triples using the pixel itself and two neighborhoods from the other two exemplars • Select the triples producing the smallest crossbar error • To speed up the process • Extract S most-similar pixel strips from each of the two exemplars, using ANN library • Form S2 triples then take 100 best triples • S = 65

12. Triples of coherent candidates • Check whether a candidate may form coherent patches in all directions with candidates from neighboring pixels • For each coordinate within a candidate triple • Verify that at least one candidate from a neighboring pixel has a continuous coordinate

13. Triples of coherent candidates Ex Ey Ez p p+1

14. Triples of coherent candidates • xC – Candidates for Ex • xCk[p] – k-th candidate triple for pixel p in Ex • xCk[p]y – Ey coordinate of the triple xCk[p]

15. Triples of coherent candidates • Iterate until having no more than 12 candidates per pixel • Typically requires 2 iterations • If more candidates remain select first 12 with the smallest crossbar error • It is possible to have no candidate at all • Rare in practice

16. Candidate Slab • Candidates are not only useful for neighborhood matching, but also provide a very good initialization for the synthesis process • For each pixel • One 2D neighborhood is in the plane of the exemplar • Two others are orthogonal to it and intersect along a line of neighborhood size of N voxels

17. Candidate Slab • To initialize synthesis we create such a slab using the best (first) candidate at each pixel • Using the slab directly as a 3D exemplar would be very limiting • This would ignore all other candidates • Uses a slab only for initialization

18. Parallel Solid Synthesis • Extended ‘Parallel Controllable Texture Synthesis’ [SIGGRAPH 2005] • Same overall structure • Upsample • Jitter • Correction

19. Parallel Solid Synthesis • Contrary to the original scheme we perturb the result through jitter only once, after initialization • If finer control is desired, jitter could be explicitly added after each upsampling step

20. Initialization • To reduce synthesis time, multi-resolution synthesis algorithms can start from an intermediate level of the image pyramid • A good initialization is key to achieve high-quality synthesis • We simply choose one of the candidate slabsand tile it in the volume • Three levels above the finest (Maximum Level L – 3) • Using the candidate slab from the corresponding level

21. Candidate Slab Random Initialization Slab Initialization

22. Jitter • To explicitly introduce variations to generate variety in the result • Perturb the initial result by applying a continuous deformation, similar to a random warp

23. Jitter • J – Jittered Volume • v – Voxel coordinate • ci – Random point in space • di – Nomalized Random direction • G = 200 • Ai = 0.1 ~ 0.3 • σi = 0.01 ~ 0.05

24. Jitter • It is important for Ai to have larger magnitude with smaller σi • Adds stronger perturbation at small scales, while adding subtle distortions to coarser scales • Small scale distortions are corrected by synthesis, introducing variety • The overall magnitude of the jitter is directly controllable by the user

25. Upsample • Each of the eight child volume cells inherits three coordinates from its parent, one for each direction

26. Correction • Performed on all synthesized voxels simultaneously, in parallel • We compute a color by averaging the corresponding three colors from the exemplars • We visit each of its direct neighbors, and use the stored coordinate triples to gather the candidate sets

27. Candidate Set • Px – 3 x 2 matrix transforming a 2D offset from Ex to volume space

28. Correction • Search for the best matching candidate by distance between voxel neighborhood and 3D candidate • Distance is measured by L2 norm on color differences • Can use PCA projection to speed up the process • Replace the triple with best matching candidate • triples have been pre-computed and optimized to guarantee that the color disparity between the three colors in each voxel is low • Two correction pass for every level • Using sub-pass mechanism of PCTS • 8 pass sub-pass

29. Correction • We gather 12 candidates from the 33 = 27 direct neighbors, for a total of 324 candidates per voxel • Too many candidates • Search for the best matching 2D candidates in each of the three directions then gather the 3D candidates only from these three best matching pixels • Still a lot • In practice we keep 4 2D and 12 3D candidates per exemplar pixel at coarse levels • 27×4 = 108 2D candidates • 3×12 = 36 3D candidates • 2 2D and 4 3D candidates at the finest level

30. Lazy Subset Synthesis • Determine the entire dependency chain throughout the volume pyramid from a requested set of voxels to synthesize the smallest number of voxels • Compute a synthesis mask • Masklp⊗Neighborhood Shape - dilation of the mask by the shape of the neighborhoods

31. Lazy Subset Synthesis • To compute a single voxel, with N = 5, 2 passes and synthesis of the 3 last levels, our scheme requires a dependency chain of 6778 voxels • The size of the dependency chain grows quadratically with the number of passes

32. Implementation and Results • Entirely in software and using the GPU to accelerate the actual synthesis • Intel Core2 6400 (2.13GHz) CPU and an NVIDIA GeForce 8800 Ultra • We sometimes add feature distance

33. Candidate pre-computation • Most results in the paper are computed from a single example image repeated three times • Pre-computed candidates may be shared depending on the orientation chosen for the image • Typically 7 seconds for 642 exemplars • 25 to 35 seconds for 1282exemplars • Includes building the exemplar pyramids, computing the PCA bases and building the candidate sets • 231KB memory required for a 642 exemplar

34. GPU implementation • Implemented in fragment shaders, using the OpenGL Shading Language • Unfold volumes in tiled 2D textures, using three 2-channel 16 bit render targets to store the synthesized triples • Pre-compute and reduce the dimensionality of all candidate 3-neighborhoods using PCA, keeping between 12 and 8 dimensions • Keep more terms at coarser levels since less variance is captured by the first dimensions • Quantize the neighborhoods to 8-bits to reduce bandwidth • Stored in RGBA 8 bit textures

35. GPU implementation • In order to minimize memory consumption, we perform synthesis into a TileTreedata structure • LEFEBVRE S., DACHSBACHER C. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D • Graphics and Games (2007)

36. GPU implementation • When interactively cutting an object, synthesis occurs only once for the newly appearing surfaces • Tile-Tree cannot be updated interactively • Store the result in a 2D texture map for display • Our implementation only allows planar cuts • new surfaces are planar and are trivially parameterized onto the 2D texture synthesized when the cut occurs

37. Full volume synthesis and comparisons

38. Full volume synthesis and comparisons

39. Full volume synthesis and comparisons • 7.22 seconds for synthesizing the 643volume from 642 exemplar • 7 seconds for pre-computation and 220 milliseconds for synthesis • Memory requirement during synthesis is 3.5MB • 28.7 seconds for synthesizing the 1283volume from 1282exemplar • 27 seconds for pre-computation and 1.7 seconds for synthesis • ‘Solid texture synthesis from 2d exemplars’ [SIGGRAPH 2007] takes 10 to 90 minutes

40. Solid synthesis on surfaces

41. Solid synthesis on surfaces • 4.1 seconds (dragon) to 17 seconds (complex structure), excluding pre-computation • Storage of the texture data requires between 17.1MB (statue) and 54MB (complex structure) • The equivalent volume resolution is 10243 which would require 3GB • Slower than state-of-the-art pure surface texture synthesis approaches • But inherits all the properties of solid texturing

42. Solid synthesis on surfaces • On demand synthesis when cutting or breaking objects (Fig. 10) • Resolution of 2563 • Initially requires 1.3MB • The average time for synthesizing a 2562 texture for a new cut is 8 ms • Synthesizing a 2562 slice of texture content requires 14.4MB • Due to the necessary padding to ensure spatial determinism

43. Comparison with a simple tiling

44. Comparison with a method using standard 2D candidates

45. Comparison with a method using standard 2D candidates • We also implemented our synthesis algorithm using only standard 2D candidates • Takes roughly twice the number of iterations to obtain a result of equivalent visual quality • Due to the increased number of iterations, the size of the dependency chain for computing a single voxel grows from 6778 voxels with 3D candidates to 76812 voxels with 2D candidates • A factor of 11.3 in both memory usage and speed

46. Limitations

47. Conclusion • A new algorithm for solid synthesis • with the unique ability to restrict synthesis to a subset of the voxels, while enforcing spatial determinism • Synthesize a volume from a set of pre-computed 3D candidates • GPU implementation is fast enough to provide on demand synthesis when interactively cutting or breaking objects