State of the Art in Global Illumination for Interactive Applications and High-Quality Animations

1. State of the Art in Global Illumination for Interactive Applications and High-Quality Animations C. Damez, K. Dmitriev, and K. Myszkowski Max-Planck-Institut für Informatik

2. Motivation • Global illumination in dynamic environments • Important in many practical applications • Increasing CPU and GPU power opens new prospects for this field • Algorithms, designed for dynamic environments • Exploit temporal coherence: • avoid redundant computation • remove temporal aliasing • Error metrics, which consider temporal aspects

3. Goal • Classify global illumination techniques in animated scenes • Recall and discuss the most up-to-date developments in this field • Evaluate respective performances, pros and cons of the new developments • Focus • Techniques supporting dynamic environments • in particular geometry changes are important for us • General purpose lighting simulation algorithms

4. Overview • Interactive methods • Radiosity (software and hardware solutions) • Stochastic techniques • Hybrid techniques • Render and shading cache • Hardware-supported high quality display methods • Off-line methods • Radiosity • Stochastic techniques • Perception guided techniques • High quality rendering methods – final gather techniques

5. Global Illumination for Interactive Environments • Interactive use (e.g. design): • Movement of objects isn’t known a priori • Geometric and material properties can change • Aiming at fast feedback to user change: • A minimum frame rate should be ensured • Image quality/precision can be traded for faster response time

6. Instant Radiosity [Keller’97] • Coarse diffuse radiance approximation is received via photon shooting • Each photon hit is treated as a point light • OpenGL rendering with shadows is used to sum up contributions from all lights • Walkthrough is possible by subtracting outdated contributions and adding new ones Images : Keller et al.

7. Discussion • Relies exclusively on graphics hardware to compute indirect illumination • fast • View dependent • camera motion causes recomputation • Recomputation for dynamic scenes is not guided by importance • Use of graphics hardware restricts the number of supported effects

8. Hybrid Rendering [Udeshi’99] • Hardware is used to compute shadows from point and area lights • One-bounce diffuse interreflections are approximated by virtual lights • CPU computes reflections & refractions Images : Udeshi et al. • 1-10 fps on computer with several graphics pipelines

9. Selective Photon Tracing [Dmitriev’02] • Indirect illumination is computed for each vertex via photon shooting • Computation is prioritized by perceptual importance of illumination update • Direct illumination is computed by graphics hardware, including shadows and goniometric diagrams

10. Algorithm Strategy (1) Initial solution is received using a fixed pool of photon paths Scene changes perturb some of those paths, which cause changes in illumination

11. Algorithm Strategy (2) • Find important paths by sparsely sampling the space with pilot photons • Select similar paths from the photon pool, using periodicity of the Halton sequence

12. Discussion • Frame rate 1-8 fps, but full illumination recovery can take up to 10 sec • View independent • camera motion doesn’t cause recomputation • No adaptive geometry subdivision • lighting artifacts (e.g. leaks) are possible • Use of graphics hardware restricts the number of supported effects

13. Distributed Ray Tracing API [Wald’02] • Instant Radiosity approach is extended by using fast ray tracing instead of graphics hardware • Caustic photons are shot to simulate caustics • Computation is distributed over a cluster of PCs, interleaving images from different clients • Discontinuity buffer heuristic is used to filter out the Monte Carlo noise Images : Wald et al.

14. Algorithm Strategy (1) • Each PC is assigned a pixel from the tile • Each PC uses own radiance representation and own set of caustic photons Image Image tile

15. Algorithm Strategy (2) • Heuristic, based on normals and distances, is used to filter irradiance values from different clients • Indirect diffuse, caustic and direct illumination is involved into the filtering process Images : Wald et al.

16. Discussion • Frame rate 1-10 fps, but computing final image can take up to 2 seconds • Wide range of effects is supported (reflections, refractions, caustics) • There is no outdated illumination • Involves huge computational power (PC cluster is required) • Image flickering is observed until enough samples are accumulated after scene or camera change

17. Incremental Update of Radiosity • First attempts at interactive global illumination based on finite element methods : • Progressive Radiosity [Chen’90,George’90] • Hierarchical Radiosity [Forsyth’94,Shaw’97] • Limited to very simple scenes • Quality/time tradeoff difficult to tune

18. Line-Space Hierarchy [Drettakis’97] • Based on Hierarchical Radiosity with Clustering • Starts from a static solution • Augments HR link structure with shafts • Hierarchical test dynamic object vs shafts p q

19. Line-Space Hierarchy [Drettakis’97] • Based on Hierarchical Radiosity with Clustering • Starts from a static solution • Augments HR link structure with shafts • Hierarchical test dynamic object vs shafts p q

20. Discussion • Time/quality tradeoff : • max # links to refine • max # links to update • Always computes a consistent solution • Memory intensive, but the storage of shafts can be avoided [Schoeffel’99] Images : Drettakis et al.

21. Unified Hierarchical Algorithm [Granier’00] • Global illumination with glossy surfaces: • Diffuse lighting: Hierarchical Radiosity • Glossy effects: Particle tracing guided by the link hierarchy • Shafts bound particles transfer • Particles are stored in mesh or in textures • Specular paths to the eye are computed as a post process

22. Interaction with the UHA • Framework similar to the Line-Space Hierarchy • Identify particles paths affected by motion: • Construction of an octree subdivided around the dynamic object position • Links are inserted in all cells they traverse • # particles resent during motion is limited by the user Images : Granier et al.

23. Discussion • Allows quasi-interactive updates of global illumination solutions (~0.5 fps) • Degraded quality during interaction to maintain a reasonable framerate • Requires a static solution to be computed beforehands Images : Granier et al.

24. Render-Cache : Principles[Walter’99] • Decouples global illumination computations from image rendering (frameless rendering) • Reconstructs the illumination from a sparse set of samples in image space • Purely software approach based on point reprojection and adaptive sampling

25. Image Reconstruction • Points in cache are reprojected on screen • After reprojection, occluded points are removed by a depth-culling heuristic • Holes filled by interpolation and filtering Images : Walter et al.

26. Choice of Samples New samples are computed asynchronously A grayscale priority image indicates regions most needing sample Dithering is used to obtain sample positions Images : Walter et al.

27. Discussion • Designed for ray tracing but can also be used with path tracers • Can be used to display specular surfaces in finite element solutions (e.g. UHA) • Scalable with high image resolution if carefully implemented [Walter’02] • Rapid movements may cause distracting artifacts

28. Shading-Cache: Principles[Tole’02] • Perform reconstruction in 3D space • avoids holes and occlusion errors • Cached samples are used to update and refine a 3D mesh progressively • Display using the graphic pipeline • high framerate • correct geometric representation of objects • temporary shading inaccuracies

29. Sampling & Mesh Refinement • Priority image is built using difference in color between vertices of a given patch • used as pdf to randomly select patches for update or refinement • To limit the size of the mesh, patches moving out of the fov are removed Images : Tole et al.

30. Discussion • Requires ~10x less samples than the Render Cache to build an image • much faster updates with a global illumination sampler • Framerate ~40-60 fps but shading update may require ~10s • manipulating objects easier and more intuitive Images : Tole et al.

31. Hardware Supported Display of Specular Effects • Mirror reflections for planar surfaces [Diefenbach’94, Wojdala’94] • Separate rendering pass for each mirror • Recursion of the basic algorithm for the higher order reflections • Refractions [Diefenbach’94] • Snell’s law: non-linear transformation difficult to implement in hardware • Approximation by the tangent law • Reflections for curved surfaces: environment maps [Green’86, Haeberli’93, Voorhies’94] • Approximated reflections • No motion parallax

32. Hardware Supported Display of Glossy Effects • Pre-filtered environment maps: For a given reflectance function store lighting reflected from the whole environment for all viewing directions • Diffuse reflectance [Greene’86] • Phong reflectance (with a Fresnel term for plastic appearance) [Heidrich’99] • Arbitrary BRDFs with rotationally symmetric lobes [Kautz’00] • Homomorphic Factorization: separation of arbitrary BRDFs into 2D textures • point and directional light sources [McCool’01] • full global illumination [Latta’02] (isotropic BRDFs only) Images: Kautz et al.

33. Efficient Prefiltering of Environment Maps • Problem: Dynamic scenes require frequent update of prefiltered environement maps • Hierarchical prefiltering [Kautz’00] – software • Hardware accelerated prefiltering [Kautz’00] • Support for shift-invariant filters of only limited size • Phong reflectance can be processed but the map must be shrunk to match the filter size • Spherical frequency space analysis [Ramamoorthi’01a] • Irradiance environment maps - Lambertian-like surfaces [Ramamoorthi’01b] • Spherical harmonics reflection maps – surfaces with complex isotropic BRDFs [Ramamoorthi’02]

34. Using Environment Mapping for Interactive Global Illumination • Irradiance Volumes [Greger’98] • revisited [Mantiuk] • Divide 3D space into a grid of volumes • For each bounce of interreflection • For each node of the grid render a cube map and compute spherical harmonic coefficients representing its irradiance [Ramamoorthi’01] • For each vertex of any object interactively manipulated by the user interpolate its irradiance based on the neighboring nodes • Asynchronously render and update irradiance at the nodes Images: Mantiuk et al.

35. Precomputed Radiance Transfer Function [Sloan’02] • Real-time rendering in dynamic, low-frequency lighting environments • Self-shadows and self-interreflection effects • Possible extensions to neighboring objects: caustics and soft shadows • Basic algorithm • Preprocessing: Computing radiance transfer function (RTF) for each rigid object • At run-time for each object • Projecting incident radiance to spherical harmonics and applying to the RTF • Convolving self-scattered radiance with the object’s BRDF and evaluating at the view-dependent reflection direction • Precomputed • Efficiently handles object self-shadows and self-interreflection effects Images: Sloan et al.

36. Overview • Interactive methods • Radiosity (software and hardware solutions) • Stochastic techniques • Hybrid techniques • Render and shading cache • Hardware-supported high quality display methods • Off-line methods • Radiosity • Stochastic techniques • Perception guided techniques • High quality rendering methods – final gather techniques

37. Global Illumination in Animations • Aiming at higher quality animations: • Constant image quality should be ensured • Popping artifacts should be avoided • Computing time is not constrained • Benefit from trajectories knowledge

38. Space-Time Hierarchical Radiosity [Damez’99] (1) • Aims at computing diffuse global illumination solutions for animations • Based on Hierarchical Radiosity • Radiosity is a function of space & time • New radiosity equation: • Formally equivalent to the static equation

39. t1 t0 t1 t0 t1 t2 t0 Space-Time Hierarchical Radiosity [Damez’99] (2) • Hierarchical solution : space-time mesh • Mesh element is polygon x time interval • Space or time subdivision • Can also be applied: • Clustering • Wavelet-basis for the time dimension

40. Discussion • Refinement is the key issue • Memory consumption is high • Performs better on “long” sequences and complex scenes (speedup = 7x-12x)

41. Multi-Frame Lighting Method [Besuievsky’96] • Global Monte Carlo method for animated scenes • Based on global multipath Monte Carlo algorithm [Sbert’96] • All positions of moving objects are inserted in the scene • Use the same set of global lines for the whole animation

42. Discussion • Reused global lines ensure temporal coherence • Speedup = 7x-25x • Adaptive meshing is not possible yet • Dense meshes increase noise and memory consumption • no high frequency details Images : Besuievesky et al.

43. Image-Based Global Illumination Framework [Nimeroff’96] • “View Space” spanned by the base range images • Key camera positions inserted adaptively based on changes in object visibility sampled by the item buffer • Visibility of reflected/refracted objects ignored • Global illumination changes sparsely sampled in time • Adaptive time steps for updating illumination • Asynchronous processing of direct and indirect lighting • Rendering through image warping and lighting interpolation between keyframes

44. Discussion • Free design of camera path within view space • Good approximation of global illumination for the whole dynamic sequence • Reduced temporal aliasing due to illumination interpolation • On current graphics hardware interactive implementation could be possible • Tone mapping not included • But the problem could be solved with HDR images • Excessively conservative computation • Direct and indirect lighting patterns can wash out each other while are processed asynchronously

45. Perception Guided Animation Rendering - Principles • Goal: take into account characteristics of the Human Visual System to concentrate the image computation exclusively on the visible scene details. • Perception-based computation guidance • Early stages of the visual path well understood • Video and animation quality metrics (AQM) are available • Higher level perceptual and cognitive models poorly elaborated • Visual attention models available, but possibly not very robust

46. Animation Quality Metric (AQM) • Computes the map of visible differences between two input animation frames • HVS modeling scope: • Weber law • Spatio-velocity Contrast Sensitivity Function • Visual masking

47. Spatio-Temporal Density Estimation Particle Tracing • View-independent light source photon tracing

48. Spatio-Temporal Density Estimation Particle Tracing • View-independent light source photon tracing

49. Spatio-Temporal Density Estimation Particle Tracing • View-independent light source photon tracing

50. Spatio-Temporal Density Estimation Particle Tracing • Extending photon processing into the temporal domain