Mixed-Order Compositing for 3D Paintings

1. Mixed-Order Compositing for 3D Paintings Presented by Hyeong Ryeol Kam 2012/05/30 Korea University Computer Graphics Lab.

2. Contents • 0 Abstract • 1 Introduction • 2 Related Work • 3 Method • 4 Discussion • 5 Conclusion

3. Abstract • A method for rendering 3D paintings by compositing brush strokes embedded in space • To reconcile between 1) the depth order (z-order in 3D) • Closer strokes over earlier strokes 2) the stroke order • Strokes later over strokes earlier - With maintaining temporal and spatial coherence

4. 1 Introduction • 2D  3D • Using stroke path, a brush stamp texture • A Technical dilemma : Order • 2D : The stroke order • To obscure all previous paint strokes • 3D : The depth order • Stroke closer obscures stroke farther away • The same surface in stroke order • The different surfaces in depth order • Using a depth tolerance, d

5. 2 Related Work • 2D • Paint By Number (1990) • Brush stroke + color + direction • 3D • Meier (1996) • Brush stroke with particles • + Color + direction + the size of brush strokes • 3D  projection to the screen plane • Disney’s Deep Canvas 3D painting system (2001) • Brush strokes – manually applied

6. 2 Related Work • 3D • WYSIWYG NPR (2002) • Not using depth order • Popping • Restricted for 3D • Keefe et al. (2001) • Strict depth order • Not aesthetic

7. 2 Related Work • The alpha channel • Catmull and Smith (1995) • Digital compositing • Wallace (1981) • The equation for the over operator • Porter and Duff (1984) • Premultiplied alpha • Mammen (1989) • Sorting algorithm (order = important, ∵ not commutative) • McCann and Polard (2009) • The user controls the order

8. 2 Related Work • Conflicting orders of compositing • Bruckner et al. (2010) • Depth order  user-specified layer order • Orders cannot change continuously • Expensive

9. 3 Method3.1 Compositing Background • The over operator • Color : h / Opacity : α • c = h α • β = 1 – α • : associative / not commutative

10. 3 Method3.2 Desired Properties • Mixed-order compositing • Should work independently on each pixel • “Single Pixel” • C(f1,f2,…,fn) • fi(ci,αi,zi,si) • n : the # of fragments at the pixel • zi : depth (starting with the closest to the viewer) • Q. zi ( ? ) zi+1 • si : stroke number : 1~n • Q. Possible that two fragments have the same stroke number?

11. 3 Method3.2 Desired Properties • Goals • Fragment close in depth • Q. Should be composited in (stroke/depth?) order • Fragment further apart • Q. Should be composited in (stroke/depth?) order • Spatial and temporal coherence • Should be maintained  Strove to balance generality But didn’t attempt to formulate a set of properties

12. 3 Method3.2 Desired Properties • To composite fragments at the same depth in stroke order • Same depth = same surface

13. 3 Method3.2 Desired Properties • Different surfaces  composite in depth order • A distance d

14. 3 Method3.2 Desired Properties • A stroke (whose alpha smoothly fades to zero) towards its borders can cause sharp visible edges • A fully transparent fragment have no effect to avoid sharp visible edges

15. 3 Method3.2 Desired Properties • During animation • Colors, α, depth may change • Popping should be avoided

16. 3 Method3.3 Existing Techniques

17. 3 Method3.4 Compositing Function • Main idea • To replace the color of each of the fragment 1) Composite nearby fragments in stroke order 2) Composite the fragments with replaced colors in depth order • Simple • But ensure continuity and good performance

18. 3 Method3.4 Compositing Function • The user specifies a global constant, d • Fragments farther than d apart • Only composite in depth order • S(z) = ( Sc(z), Sα(z) ) • The result of compositing all fragments with depths strictly between z-d/2 and z+d/2 in stroke order • No fragment  S=(0,0) • S is a piecewise-constant function with discontinuities at zi+d/2 and zi-d/2 • Compute the colors and alpha as:

19. 3 Method3.4 Compositing Function • Because the colors are premultiplied with alphas, this integral is correctly weighted by alpha • We replace the fragment colors, while keeping their original alpha values, setting c’’i=c’iαi/α’i • The final output : • Not exhibit discontinuities

20. 3 Method3.5 Algorithm

21. 3 Method3.5 Algorithm • 1. Sort : fragments (by stroke number 1~n) • 2. S is modified to include the new fragment • [zi - d/2 , zi + d/2]  Complete binary tree

22. 4 Discussion4.1 Implementation and Results • Test 3D paintings drawn in OverCoat • OverCoat generates splats by projecting stroke centerlines onto the 2D display and uniformly sampling them in 2D

23. 4 Discussion4.1 Implementation and Results

24. 4 Discussion4.2 Parameters and Temporal Coherence • d (window size) • Should be large enough • Sometimes useful to vary d with depth • Wide brush strokes may need to be composited in stroke order over a larger window than thin brush strokes

25. 4 Discussion4.2 Parameters and Temporal Coherence • r : 0.5 • Smaller value decreases the amount of blending

26. 4 Discussion4.3 Extensions, Limitations, and Future Work • Extensions • Large scenes & d << the depth range • The running time : O(n log m) • m : the maximum number of fragments in an interval of length d • Can be sped up using a dynamic binary tree • Memory usage : O(m) • Limitations • The surface pass through each other

27. 4 Discussion4.3 Extensions, Limitations, and Future Work • Future Work • The stroke order is irrelevant and only a temporally-coherent depth-order compositing that satisfies Properties 2-4 is needed • Re-define :

28. 5 Conclusion • 3D paintings • Great potential • To allow to make use of 3D structure • Over limitations of the old 3D pipeline • A sound rendering & compositing method • Necessary for animated scenes • To gain wider use and realize the potential