Paint By Numbers

1. Paint By Numbers • The goal of a visual artist (Hagen): • Without modeling detail, painters use brush strokes to: • Represent objects • Direct attention The goal of effective representational image making, whether you paint in oil or in numbers, is to select and manipulate visual information in order to direct the viewer’s attention and determine the viewer’s perception.

2. Basic Approach • Aim: To take a synthetic or natural scene and convert it into an abstract representation • The paper describes an interactive tool • A user drags the mouse over a source image • The image is point sampled, and the color drives a brush stroke • Avoids explicit color selection common in paint programs • Controlling the nature of the strokes controls the form of the picture

3. Representation • A painting is an ordered list of brush strokes • A brush stroke is described by: • Location • Color • Size • Direction • Shape • The aim is to use the strokes to convey: • Surface color • Surface curvature • Focus • Edges

4. Controlling the Brush Strokes • Position comes from mouse with random noise added • Color comes from source image, with random noise added • Size comes from the speed of mouse motion or explicit user selection • Stroke direction comes from mouse motion direction or explicit user selection • Shape is chosen by the user (circles, rectangles, lines, scatterings, user design)

5. Post Processing Operations • All brush strokes can be stored and later manipulated • Operations include: • Add noise to some component • Simple animation by applying same stroke parameters to sequence of images, choosing color from each image • Use gradient to set orientation • Use second derivative to enhance edges

6. Some Other Things • Use ray tracing to extract color from a 3d model • Apply simple stochastic gradient decent to automatically place strokes • Fix the number of strokes • Repeatedly: Perturb an attribute, of the result is better keep the change • Using cone shaped brushes produces Voronoi diagrams (cute observation, completely unrelated to NPR)

7. Computer-Generated Pen-and-Ink Illustration • Illustrations offer many benefits over traditional rendering: • Detail can be enhanced, or omitted • Vitality and style • Applications: • Medical illustration, Workshop manuals, Design sketches • Pen-and-Ink Features: • Easy and cheap to print, and goes well with text • Color and shading must be implied by stroke combinations

8. Strokes • A curve segment traced out by a pen • Principles: • Too thin is washed out; too thick obscures detail • The pen may be turned or varied during the stroke • Strokes should look natural - to add life. Characteristics, such as width, should be varied • Wavy lines indicate immature ideas (sketches)

9. Tones and Texture • Combinations of strokes combine to form: • Tone: the brightness of a point • Texture: an indication of the surface properties • Principles: • Roughly equal weight and spacing for tones • Absolute values of tones is not really important, but differences in tone are very important • Tones should sometimes be forced to disambiguate objects

10. More Tones and Textures • Texture principles: • The character of strokes is important for texture • Crisp straight lines for “glass” • Horizontal surfaces with horizontal lines • Absence of detail indicates glare • Sketchy indicates old, crisp indicates new • To lend economy to the image, texture should be indicated: • Only draw some subset of strokes • Vary the exact method across the illustration

11. Outlines • Outlines are completely abstract - photographs don’t have them • Not just for contours - may also indicate interior detail (veins in a leaf) • Principles: • Outline supports textures • Thick lines for shadows, or less depth • Haloes where objects pass behind each other • Outlines where tone is omitted to convey shape • Indication is just as important as it is for texture

12. Input and Preprocessing • A 3D model • Associate texture with each face • Render with diffuse+specular to get reference tones • Indicate where detail should appear • Indicate where the light is • Generate shadow polygons using a BSP tree shadow volume algorithm • Generate a map of the screen, indicating what appears where, and the relative depths

13. Laying Down Tone and Texture • Fill each region with “stroke texture”: • Set of strokes with priorities as to which should be rendered first • Scale independent - appropriate strokes get drawn depending on resolution and size • User indicated detail area drive placement • Field describes how much texture should be drawn • Field drops off away from detail edges • Strokes are clipped to regions, with some noise

14. Laying Down Outlines • Outline associated with each texture • If shading indicates an edge, do not drawn outline • Where surfaces meet, use closest texture as outline • Thick or thin lines to indicate shadow and relation to the light sources • Dependence on viewing direction: • View direction affects which strokes are drawn

15. Results

16. Still There to Solve: • Automatic emphasis control • Rendering other types of scenery (such as trees, to be presented later) • Animating pen-and-ink • Coherence • Adding and dropping strokes as priorities change

17. Automatic Technical Illustration The underlying assumption in NPR is that artistic techniques developed by human artists have intrinsic merit based on the evolutionary nature of art. • Concentrates on shading, assuming edge lines are desirable and will be added • Key techniques: • Cool-to-warm shading • Metal shading

18. Traditional Shading • Assume highlights and edge lines are also added • In shadow areas, ambient term dominates: • Ambient must be high to contrast edge lines • No detail in shadow areas • In areas facing light source, brightest diffuse color dominates • No contrast with highlights • Have to reduce kd • Detail is lost!

19. Tone-based shading • “Tones” come from adding gray to a color • Shifts hue, but leaves luminance roughly intact • “Temperature” of a color is: • High if color is close to red • Low if color is close to blue • Hot colors appear to be forward, while cold colors appear to be back • Algorithm: • Replace cosine term in regular model with a new term that blends between hot and cold

20. New Model • Regardless of base color, resulting tones will be cold or warm • Then use: • Variation even in shadow regions • Robust to parameter settings

21. Metal Shading • Traditional methods indicate metal by alternating dark and light bands • Bands are aligned with direction of minimum curvature (so along the axis of a cylinder) • Derived from anisotropic reflection of milled objects • Modeled by adding 20 stripes around the object, and a bright stripe closest to the light source

22. Real-Time Approximations • Aim: To approximate the cool-to-warm shading model with standard rendering hardware • Use two lights to get cool-to-warm • Warm light from one direction, cool light from opposite direction • Exploits negative intensity light sources • Stencil buffers or software to get contours and silhouettes • Texture or environment maps to get metal shading • Rotate the object, keeping the viewer and light fixed

23. More Open Problems • They do not exploit cool-to-warm to help with depth perception. Can that be done? • Related to another problem: Contrast maintenance/enhancement in tone reproduction algorithms • Basic idea: Adjust the shading to enhance local features, while maintaining some global consistency