Integration Of CG & Live-Action For Cinematic Visual Effects

1. Integration Of CG & Live-Action For Cinematic Visual Effects by Amarnath Director, Octopus Media School

2. INTRODUCTION • Visual Effects (VFX) have served an increasingly important role in film since the late 1980s as the computational power provided by enhanced hardware and software has rapidly increased • Numerous films have demonstrated the art of invisibly bridging the imaginary and the real with the aid of computer graphics • It is possible to produce any kind of visual feast imaginable with computer-based technologies - Photorealistic actors, Dinosaurs, Aliens, Transforming Robots – you name it

3. The key to achieving realism in much of visual effects is to successfully combine a variety of different elements - matte paintings, locations, live-action actors, real and digital sets, CG characters and objects - into a single shot that looks like it was all there at the same time. • An important, subtle, and frustratingly complex aspect is to match the lighting amongst these elements. Not only do the objects and environments need to be lit with the same sources of light, they need to properly reflect and shadow each other

4. For successful compositing it is crucial to seamlessly integrate real and CG components. • Such a seamless integration requires solving a set of extremely difficult problems. • Most of these problems are related to recreating the following real world information: • Camera positions and parameters • Illumination information to recreate the same lighting environment for both CG and real objects

5. Workflow for Integrating Live-action and CG • Shooting • Matchmoving • CG Integration • Image Based Lighting • Rendering CG • Compositing • Color Grading

6. Shooting Match Moving CG Integration Image Based Lighting Rendering Compositing Grading Output

7. Shooting • When shooting a scene that will use CGI or 3D Animation interacting with live action there are many precautions to be taken to ensure that you get what you need to make the shot work

8. Recording Camera Settings • By recording the position and focal length of camera during the shoot it allows the CG Artist to match the virtual camera with the field camera making the perspective of shot look accurate • Photo Reference • Photos of surroundings to use in reflections • Photos of reference objects • Green Screen • For situations where CG characters and real actors crossover

9. Matchmoving • Matchmoving is one of the most crucial processes in visual effects for integration of CG elements into live action footage. Matchmoving enables insertion of CG elements into live action footage so that the position, scale, orientation and motion of the CG elements are relative to the shot footage. • The process of matchmoving involves generating a virtual camera by tracking the movements of a camera through a live-shot scene, so that the tracked virtual camera can be used inside 3-D programs to match CG elements to the shot footage.

10. Process of Matchmoving • On Set - • camera information (camera height and angle, focal length) • set measurements • survey data (distance between camera and objects, distance between track points, etc.) • Matchmoving Software • Feature Tracking • Camera Solve • Exporting the camera for 3d software

11. CGI Integration • Based on the camera solve information exported from the matchmoving software, CG elements (Set Extension, 3D Character, Building, Vehicles etc.,) are placed at the right position. • Scale and perspective of the CG elements are also matched to the live-action plate. • The CG elements are modeled, textured, and lit to match the scene lighting

12. Lighting • The CG elements are lit to match the scene lighting using lights in the 3D program and as well as through Image Based Lighting • Image based lighting generally creates softer shadows, so its required to use the lights in the software to create better and sharper shadows

13. Image Based Lighting • In the context of visual effects, the advent of image-based lighting (IBL) has made a significant contribution to providing real-world lighting and rendering CG objects • IBL has tremendously facilitated the process of lighting-setup in standard 3D softwares in terms of raising the believability & photorealism of CG and live-action integration

14. Process of Image Based Lighting • Images of light in the real world are obtained in various ways • Fish Eye Lens (gives a 180 degree field of view) • Mirror Ball or Light Probe (Photographs of varied exposures of a mirrored ball placed in a real scene for catching the reflected view on its surface)

15. Light Probe Fish Eye Lens

16. HDR (High Dynamic Range) Images are generated from the photographs taken from any of the above method • The HDR image is then projected as an environment map in the 3D Software and used as the source of illumination through the aid of a global-illumination rendering process to provide the lighting matched to the original environment for CG objects and scenes, • Hence this process is named Image Based Lighting

17. Rendering • Once the CG elements are complete with all the details to make it look seamless in the live-action plate, the scene is rendered as an image sequence with various passes (render passes) • The render passes allows the user to make further corrections of the output in compositing software without the need to re-render the scenes

18. Render Passes • The most common passes that are rendered are • Beauty Pass or Diffuse Pass • Specular Pass • Reflection Pass • Shadow Pass • Depth Pass • AO (Ambient Occlusion)

20. Compositing • Compositing is the process of taking images from variety of sources and combining them in such a way that they appear to be shot at the same time under the same lighting conditions with the same camera. • The image sequences of the render passes are imported into a compositing software. • The sequences are then tweaked further for properties such as reflectivity, specularity etc so that they match perfectly with the live footage