Interactive Computer Graphics Visual Effects

1. Interactive Computer GraphicsVisual Effects James Gain and Edwin Blake Department of Computer ScienceUniversity of Cape Town September 2002jgain@cs.uct.ac.za Collaborative Visual Computing Laboratory

2. Objectives • To introduce the use of Computer Graphics in the Visual Effects industry • Case Studies: • Bullet Time (“The Matrix”) • Virtual Actors • Painterly Rendering (“What Dreams May Come”) • SFX Maxim: the special effect should be a servant of the story Interactive Computer Graphics Contents

3. “The Matrix” • Bullet Time: • The camera rotates while the action remains frozen or unfolds in variable ultra-slow motion • To achieve bullet time an ordinary camera would need to shoot 12000 frames per second while moving around the actor at 20 meters per second • Inspiration: "The Wachowskis [directors] are from the comic-book culture, and are therefore familiar with the Japanese animation style called anime, [which] takes advantage of 'the physics of decimation' ­ it breaks down action into its components and allows those elements to be meticulously controlled to build the most dramatic effect from dynamic movement.“ Interactive Computer Graphics Contents

4. Bullet Time: A Synthesis of Techniques • 5 different VFX techniques: • Wire removal: the actor is suspended on a wire harness, which is tracked across frames and removed relatively automatically • Blue Screening: the actor performs against a uniformly coloured screen • Flow-Mo: uses multiple cameras to achieve ultra slow motion • Morphing: used to inbetween individual frames • Virtual Sets: a synthesis of real and computer generated backgrounds replace the uniformly coloured screen Interactive Computer Graphics Contents

5. Blue Screening • Originally an optical process (also known as a travelling matte) to composite two or more pieces of film: • Shoot a background plate of the scenery • Shoot the actor(s) against a blue screen • Use a red filter to form two mattes (blackout areas). One covers the actor(s) and the other is its reverse • Cover new film with the actor silhouette matte and expose to the background plate • Rewind film, place reverse matte, expose the filmed actor • The mattes change in every frame • Can have hundreds of different layers (Star Wars) • Now Computerized. Scan film, digitally composite, create optical print Interactive Computer Graphics Contents

6. Problems with Blue Screening • Need to tweak the background and foreground so that they fit seamlessly: • Reflections – If light from the blue screen is diffusely reflected onto the actor then a blue fringe (or “bleed”) may result • Blue clothing – If clothing or skin is a similar shade to the screen then it may be matted out • Relative focus and lighting – If the background is in focus or the background and foreground are lit differently then the compositing is obvious • Attention to detail is crucial when faking reality Interactive Computer Graphics Contents

7. Camera Animation • Technique: • Actor performs the action • Still 35mm cameras arranged along a curved path capture the scene • They can be fired sequentially with variable millisecond delays (ultra-slow motion) or all at once (frozen time) • The camera position, orientation and focus are pre-visualized in a virtual environment • Once the scene is shot, the wire and cameras are removed Interactive Computer Graphics Contents

8. Postproduction • Morphing: • Once the scene is shot, morphing software inbetweens individual frames • Morphing involves interpolation between corresponding regions in two images • Early uses: Michael Jackson music video and the film “Willow” • Virtual Sets: • Shoot the background separately without trying to match-move with the camera • Instead use computer vision techniques to roughly reconstruct a 3D environment. Cover with texture extracted from the background shots • Composite virtual set with actor • Bullet Time Summary: Enables total and independent control over the speed of camera and actor motion Interactive Computer Graphics Contents

9. Virtual Actors • A Virtual Actor is the CG representation of a character’s motion and appearance • Sometimes known as a synthespian or vactor • Exercise: Why would we want to replace real with CG actors? Interactive Computer Graphics Contents

10. The Need for Virtual Actors Actor must react in real time (e.g. computer games) Actor is to be placed in a computer generated set (e.g. “Titanic”) • Cheap Rent-a-Crowd: • Actor is a member of a crowd scene • Colliseum crowd in “Gladiator” Resurrecting the Dead: • The real actor is dead or unavailable • Marilyn Monroe in “Rendezvous A Montreal” • Aliens and Monsters: • Actor’s physical form is difficult to realize • Jar-Jar Binks in “Star Wars: episode 1” • The Perfect Stuntman: • Actor must perform impossible or dangerous actions • “Spider-Man” Interactive Computer Graphics Contents

11. Challenges and Limitations • Remaining Challenges: • "The human face is a unique problem. We are genetically programmed to recognize human faces. We're so good that most people aren't even aware of it while they think about it. It turns out, for instance, that if we make a perfectly symmetrical face, we see it as being wrong. So we want things to be not quite perfect, have a lot of subtlety, but if they're too imperfect, then we think that they're strange.” – Ed Catmull (founder of Pixar) • Limitations (economic realities): • "A dinosaur doesn't exist, so it's practical to simulate it. With human beings, however, having a staff of 20 people all working on the lighting, the modeling, and the motion might not be a great trade-off, because you can replace that whole team with one human actor who can do what the director wants.” – Jim Blinn Interactive Computer Graphics Contents

12. “What Dreams May Come” • ‘Painted world’ relying on 19th C. art ranging from Germanic Romanticism to Stained Glass Impressionism. • Requirements: • Needed to allow interaction between actor and painted environment. • Didn’t want an image-based technique prone to a “shower-door” effect. Before VFX Post-Production After VFX Post-Production Interactive Computer Graphics Contents

13. VFX Process • Image Processing: • Scan in film stock (analog to digital). Soon to be outmoded? • Used orange balls to help with registration between frames • Scene also filmed with Lidar (Laser Radar) scanning • Computer vision used to reconstruct a 3D model of the scene • Computer Graphics: • Apply painterly rendering • Composite with actor Interactive Computer Graphics Contents

14. Non-photorealistic Rendering and Animation • Computer Graphics has generally striven for heightened realism • An alternative: concentrate on the aesthetics of the image • Artistic effects: loosely simulate oil or pastel paintings, or pen and ink illustrations • Achieve artistic effects: • Variations in brush stroke size, direction, colour and texture allow abstraction (downplay artistically unimportant aspects), rhythm (sense of energy), focus (emphasize artistically important aspects) and mood • But must avoid: • Introducing too much noise or losing coherence between frames of an animation Interactive Computer Graphics Contents

15. Object and Image Space • Combines object and image space algorithms • Object Space: • Particles on the object surface ensure consistency between animation frames. Avoids the “shower door” effect • Image Space: • Reference images of the scene are used to determine the orientation, colour and size brush attributes • Reference images of brush strokes are altered according to the brush attributes and composited onto the image Interactive Computer Graphics Contents

16. Painterly Algorithm Create particles to represent geometry FOR each frame of animation Create reference pictures using geometry, surface attributes and lighting Sort particles by distance from viewpoint FOR each particle (starting with furthest from viewpoint) [1] Transform particles to screen space [2] Determine brush stroke attributes from particles and/or reference pictures [3] Randomly perturb attributes based on user selected attributes [4] Composite brush stroke into paint buffer ENDFOR ENDFOR Interactive Computer Graphics Contents

17. Painterly Algorithm Interactive Computer Graphics Contents

18. Painterly Examples • Altering brush stroke and painting parameters to create different effects: pastel, pointilist, marker-pen, woodcut Interactive Computer Graphics Contents

19. Conclusions • Jittering: • Some random variation in brush attributes is needed to ensure a painterly appearance • Problem: this damages temporal coherence • Solution: associate a random seed with each particle • Painterly rendering is an intensive process which requires considerable user intervention (parameter tweaking) • May need several passes to composite different effects: shadows, highlights, etc. • Tip of the Iceberg. There has been much research in this area recently, producing a variety of techniques in both painterly and line-art rendering Interactive Computer Graphics Contents

20. Other Non-PhotoRealistic Styles • Pen-and-Ink Illustration: • Difference Image Algorithm: • Draw a hatching stroke in the darkest part of a greyscale image • Subtract a blurred version of the stroke from the image and repeat • Cartoon Illustration: • Place Graftal (procedural) textures along silhouettes • Evoke the style of Dr. Suess • Effectively indicates complexity, e.g. grass, fur, trees Interactive Computer Graphics Contents