Walk-Through Illustrations: Frame-Coherent Pen-and-Ink Style in a Game Engine

1. Walk-Through Illustrations:Frame-Coherent Pen-and-Ink Style in a Game Engine Bert Greudenberg Maic Masuch Thomas Strothotte

2. It Looks All Crappy Cartman: Aw, screw it. It probably isn't all that good anyway. Kyle: Cartman! What are you talking about?! You LOVE Terrance and Phillip! Cartman: Yeah, but the animation's all crappy — it probably can't sustain itself over ninety minutes.

3. Who wants crappy animation? • Tired of the gothic space dungeons? • Tired of “realistic” animation that fools you for five seconds? • You need a change • A non-photorealistic change!

4. 3D Game Engines • 3D game engines have made huge advances • Engines primarily still focus on photorealistic images • Game images are technically non photorealistic

5. NPR Here to Stay • Research has turned to NPR of still images • Great advances in Toon, Painterly, and Pen & Ink (i.e. South Park) • However, animating NPR or real-time rendering 2 problems occur.

6. What’s the Problem? • Maintenance of frame-to-frame coherence • Hidden Surface Removal • Objects appear and disappear correctly • Ability of a 3D engine to render NP images in real time

7. Solution! • Create an engine that facilitates a hybrid rendering pipeline • Allows specification of the rendering style on a per object basis More to follow

8. Computer Games and NPR • No games achieves true photorealism • Rendering style can be called photorealistic • Shapes, spatial relationships, surface details

9. NPR Video Games • Very few games incorporate NPR elements • Examples: • Cartoon Rendering • Pen & Ink • Broken Arrow

10. Cartoon Rendering • Shading Model in 2D cartoon • Games only use 2D data. • Visually appealing, but not that interactive • Monkey Island III • Broken Sword

11. Monkey Island III

12. Pen and Ink Rendering • NPRQuake by Gleicher • Intercepts the calls to graphics libraries • Replaces them with different drawing primitives • Uses core GLQuake engine and maintains gameplay

13. NPRQuake Sketch

14. NPRQuake Blue-Print

15. NPRQuake Brush-Stroke

16. Problems with NPRQuake • Achieves non-photorealistic look • Style elements do not preserve frame coherence • Outlines of objects vary from frame to frame • Results in lively but disturbing experience

17. How to Fix It? • Few systems deal with frame to frame coherence. • Many systems can create Pen & Ink illustrations • But, probability is used to model the line deviations of hand drawings • Not suitable for real time rendering in animations

18. Non-Photorealistic Visualization In Game Engine • NP imaging allows the user to simplify an image by omitting details to control the viewer’s attention • Emphasize, Deemphasize certain elements of a scene • Aim for pen-and-ink illustrations • Silhouettes and creases are clearly drawn • Surface detail added through hatching

19. Hybrid Rendering Pipeline • Able to specify rendering style of each object • Gain additional freedom in expression for the visualization • Photorealistic Images – use normal texture mapping and shading • NP Images – consist of modified shading, an outline, and NP textures (ink maps or hatch maps) • Hybrid – combine photorealistic rendering and NP drawing styles

20. NP Elements in 3D Game Engine • Properties of 3D game engine • Provide player with highly immersive experience • Enhace speed, realism, and image quality • Preprocess as much data as possible • Let hardware handle most of the per-frame load • Mip-Mapping, anti-aliasing

21. NP Elements in 3D Game Engine • Non-Photorealism uses image space elements • Pixels generated on the screen are not a strict projection of the 3D scene objects • A triangle outline rendered as lines has a constant width independent of the actual distance of the triangle to the viewer

22. Outlining • Silhouettes • Drawn to enhance the visual separation of objects from each other and background • Silhouette edges need to be determined every frame • Non-Convex edges are never drawn (second) • Sharp Edges are always drawn (third) • Only smooth convex edges are drawn (fourth) • Result with hidden lines removed

23. Hatching • Textures are supposed to become smaller with increasing distance • Hatching • shade by drawing or etching fine parallel or perpendicular lines • Pen & Ink hatching textures cause serious moire´e patterns • In motion this produces even worse artifacts than still images • Prevents frame coherence

24. How to Fix This? • Common antidote • Mip-mapping • Unfortunately, normal filtering process applied to large white areas and thin black lines of hatching texture cause lines to vanish • Visible structure is lost

25. Hatch Maps • Normal mip-mapping • Lines are black and well separated at the top level • Become lighter and closer at lower levels • Hatch Map • Equally space black hatch lines for all mip-map levels • Only very lowest levels are gray values used to maintain tone

26. Hatch Maps • Unfortunately, all over tone is not continuous • Abrupt switch between hatch-map levels • Use Tri-linear Filtering

27. Ink Maps • Hatching is a mechanical process • Hybrid Pipeline • Photorealistic: Mip-Mapping • Non-Photorealistic: Ink Map • Adding details to surfaces is artistically challenging • Construct ink maps • Smooth animation achieved with carefully crafted ink maps • Frame Coherence is maintained using tri-linear filtering

28. What They Achieved • Part of a medieval palace created for the UNREAL engine

29. What they Achieved

30. Specifications • 3D engine runs on: • 750 MHz CPU • GeForce2 ultra graphics card • Even on unoptimized models, engine runs at interactive speeds, i.e. 25 fps

31. And Now….Zelda!!! • The Legend of Zelda: The Wind Wakers • Right now, this is the pinnacle of NPR in game engines. • First game to max out hardware of Nintendo Gamecube • Combines great use of interactive 3D and NPR

32. Screenshots

33. More

34. Zelda Video!! Thank You Any Questions?