Non-PhotoRealistic Rendering: Evaluation and Examples

1. NonPhotoRealism and an Evaluation of Rendering Lecture 4

2. Painterly Rendering: Scene Reconstruction • Scene Data Capture (from ‘What Dreams May Come’): • Scan in film stock (analog to digital). Soon to be outmoded. • Used orange balls to help with registration between frames. • Scene also filmed with Lidar (Laser Radar) scanning. • Computer vision used to reconstruct a 3D model. Before VFX Post-Production After VFX Post-Production

3. Painterly Rendering: Process • Combines object and image space algorithms. • Object Space: • Particles on the object surface ensure consistency between animation frames. • Image Space: • Reference images of the scene are used to determine the orientation, colour and size brush attributes. • Reference images of brush strokes are altered according to the brush attributes and composited onto the image.

4. Painterly Rendering: Algorithm Create particles to represent geometry FOR each frame of animation Create reference pictures using geometry, surface attributes and lighting Sort particles by distance from viewpoint FOR each particle (starting with furthest from viewpoint) [1] Transform particles to screen space [2] Determine brush stroke attributes from particles and/or reference pictures [3] Randomly perturb attributes based on user selected attributes [4] Composite brush stroke into paint buffer ENDFOR ENDFOR

5. Painterly Rendering: Algorithm

6. Painterly Rendering: Conclusions • Jittering: • Some random variation in brush attributes is needed to ensure a painterly appearance. • Problem: this damages temporal coherence. • Solution: associate a random seed with each particle. • Painterly rendering is an intensive process which requires considerable user intervention (parameter tweaking). • May need several passes to composite different effects: shadows, highlights, etc. • Tip of the Iceberg. There has been much research in this area recently, producing a variety of techniques in both painterly and line-art rendering.

7. Painterly Rendering: Examples • Altering brush stroke and painting parameters to create different effects: pastel, pointilist, marker-pen, woodcut.

8. Other NonPhotoRealistic Styles • Pen-and-Ink Illustration: • Difference Image Algorithm: • Draw a hatching stroke in the darkest part of a greyscale image. • Subtract a blurred version of the stroke from the image and repeat. • Cartoon Illustration: • Place Graftal (procedural) textures along silhouettes. • Evoke the style of Dr. Suess. • Effectively indicates complexity, e.g. grass, fur, trees.

9. NonPhotoRealism: References • Survey Article: • Landsdown, J. and Schofield, S. “Expressive Rendering: A Review of Nonphotorealistic techniques”, IEEE Computer Graphics and Applications, Vol. 15, No. 3, 1994, pp. 29-37. • Painterly Rendering: • Meier, B. “Painterly Rendering for Animation”, SIGGRAPH ’96, pp. 477-484. • Haeberly, P. “Paint by Numbers: Abstract image representations”, SIGGRAPH ’90, pp. 207-214. • Line-Art Rendering (pen-and-ink lines and cross-hatching effects which emulate sketching): • Wikenbach, G. and Salesin, D. “Computer Generated Pen-and-Ink Illustration”, SIGGRAPH ’94, pp. 91-100. • Cartoon Illustration: • Kowalski, M. et al. “Art-based Rendering of Fur, Grass and Trees”, SIGGRAPH ’99, pp. 433-438.

10. Rendering in Computer Graphics • Principal Rendering Techniques: • Polygon Scan Conversion • Ray Tracing (Basic, Distributed, Recursive) • Adjunct Rendering Techniques: • Radiosity • Non-PhotoRealistic Rendering • Surface Detail • Evaluation: • Realism • Speed • Memory Consumption • ‘Real World’ choices

11. Realism: Direct Illumination • Light may reflect off many surfaces on its way from the light to the camera. • Standard ray tracing and polygon scan conversion can handle a single diffuse or specular bounce. • Direct Diffuse Reflection • Assumes that the object is a perfect Lambertian (diffuse) reflector. • Direct Specular Reflection • Use Phong approximation to true specular (highlight) reflection.

12. Realism: Indirect Illumination • Distributed recursive ray tracing can handle multiple specular bounces. Recursion halts when a diffuse surface is encountered. • Radiosity can handle multiple diffuse bounces. • The general case: • Some research into adapting and combining radiosity and ray tracing. • Results tend to be very inefficient. [1] Diffuse Specular [2] SpecularSpecular [3] DiffuseDiffuse [4] SpecularDiffuse

13. Realism: Effects • Motion Blur: • [RT] Temporally distributed ray tracing OR • [PSC] Blending between frames with Accumulation Buffer. • Refraction: • [RT] Recursive ray tracing of refracted rays. • [PSC] Not supported but can do non-refractive transparency. • Depth of Field (focus): • [RT] Distributed ray tracing over the camera lense OR • [PSC] Use z-values to affect focus • Other Effects: lense flare, antialiasing. • Achievable by both RT and PSC.

14. Realism: Reflection in PSC • [RT] Recursive ray tracing of reflected rays. • [PSC] Environment Mapping • Also known as reflection mapping. • Render the scene in all directions from a single viewpoint. Can render spherically or onto a box. Use these rendered images as a texture map. • Prone to distortion. A single central reflection point stands in for multiple reflection points on the object’s surface. Particularly poor for planar surfaces. • [PSC] Scene Duplication • Create a darker duplicate of the scene behind the reflective object. Only really works for reflections off a plane.

15. Realism: Shadows • Shadows depend on visibility from the light source. Surfaces not visible are in shadow. • [RT] Basic ray tracing, with rays shot to light sources. • [PSC] Methods: • Scan-line: intersect current scan-line with potential shadowing edges – very expensive. • Pre-processed surface detail: place surface detail polygons over surfaces visible from the light source. Done once per scene per light source. • Shadow volumes: The volume in shadow behind a polygon is stored. During rendering viewing rays are intersected with the shadow volumes. • Two-pass z-buffer: Combine z-buffers rendered from the light source (shadow mask) and viewpoint. Efficient but can have aliasing problems.

16. Rendering Evaluation • PSC is capable of emulating many of the effects but not illumination of RT.

17. Costs: Computation • Given: • Recursive and Distributed Ray Tracing: • ray-object intersections • Polygon Scan Conversion: • polygon scan-line renderings • Seemingly independent of , but affects screen size of polygons. • As increases the size of polygons and their scan-line rendering time tend to decrease.

18. Costs: Memory • Both algorithms require: • screen buffer (assuming 16 bits per colour component) • scene storage (assuming exclusively naïve triangle meshes ) • Recursive and Distributed Ray Tracing: • Negligible storage of bounding objects or cell occupancy lists. Otherwise no extra costs. • Polygon Scan Conversion: • Z-buffer: (assuming 32 bit depth value)

19. What is Actually Used? • Visualization: • Clarity over realism = PSC • Sometimes volumetric rendering is required. • Simulation, 3D Games: • Speed over realism = PSC + (Sprites, Ray Traced backdrops, Radiosity) • Extensive pre-processing is acceptable. • Special Effects: • Realism over speed BUT high resolution (6000x4000) = Mostly PSC + some RT for ‘beauty’ shots. • Example: Toy Story (proprietary PSC required 1-3 hours per frame). • Not all SFX are computer generated. Use Models, mattes (hand-painted backgrounds) and pyrotechnics.