Non-Linear Kernel-Based Precomputed Light Transport

1. Non-Linear Kernel-Based Precomputed Light Transport Paul Green MITJan Kautz MITWojciech Matusik MITFrédo Durand MITHenrik Wann Jensen UCSD

2. Interactive High Quality 6D Relighting Reflectance Geometry & Viewpoint All-Frequency Lighting Rendered Frame

3. Shadowing Inter-reflection Incident Radiance Precomputed Light Transport Transport function maps distant light to incident light Can Include BRDF if outgoing direction is fixed Distant Radiance Exit Radiance Courtesy of Sloan et al. 2003

4. Light Transport Linear Operator • Radiance Loat point p along direction is weighted sum of distant radiance Li Transport Vector Outgoing Radiance Distant Radiance (Environment Map)

5. Example It’s a Dot Product Between Lighting and Transport Vectors!! Transport Function (log scale) Environment Map Exit Radiance (outgoing color) BRDF Weighted Incident Radiance

6. Direct Evaluation Infeasible • Raw Transport Matrices Enormous • 50,000 Vertices • 24,200 Element Environment Map • 92 View Directions • Direct Lighting-Transport product too costly • 24,200 multiplies / vertex 50 GBs raw data But the formula still works in any other basis

7. PLT is Representation • Key issue of PLT is representation of Transport and Lighting efficiently. • Efficiency of: • Storage • Lighting-Transport Product

8. Linear Approximation • Precomputed Radiance Transfer [Sloan et al 02,03] • Low Order Spherical Harmonics • Soft Shadows • Low Frequency Lighting • Not Practical For High Frequency Lighting Courtesy of Ng et al.2003

9. Non-Linear Approximation • Non-Linear Wavelet Lighting Approximation [Ng et al 03] • Haar Wavelets • Non-Linear Approximation • All Frequency Lighting • Arbitrary BRDF -> Fixed View Courtesy of Ng et al.2003

10. What is Non-Linear Approximation? • Non-linear: Approximating basis set depends on input Non-linear: 5 Largest Coefficients Linear: First 5 Coefficients SSE = 935 SSE = 3,037

11. Overview of our method • Precompute Transport at a sparse set of sample view directions • Backwards Photon Tracing • Approximate Transport • Non-Linear Parametric Representation • Render • Fast Lighting-Transport Product • View Point Interpolation

12. Precomputation Transport In spherical coordinates (Lat/Long) Log Scale Elevation Azimuth

13. Precomputation Backwards photon tracing from fixed view , fixed position p Diffuse Specular Inter-reflections Shadowed Refracted

14. Density Estimation Photon Cloud After Density Estimation

15. Factoring Transport Full Transport View Independent (diffuse) Consistent across all views View Dependent (specular) One per view

16. Non-linear Approximation p Recap

17. p Overview of our method • Precompute Transport at a sparse set of sample view directions • Backwards Photon Tracing • Approximate Transport • Non-Linear Parametric Representation • Render • Fast Integration Method • View Point Interpolation

18. Non-linear Approximation Non-Linear Approximation • Transport approximated by sum of constant valued box functions • Arbitrary weight, size and position • Expressive as Haar Wavelets • Even More Flexible! Original wj – weight Kj – size and position Approximated

19. Non-Linear Approximation Box: Arbitrary location & size Our approach Wavelet: Rigid dyadic domain [Ng et al. 03]

20. Non-Linear Approximation Non-Linear Wavelet Approximation 5 Largest Coefficients [Ng et al.] Non-Linear Box Kernel Approximation 5 Boxes Our approach SSE = 652 SSE = 935

21. Non-Linear Approximation • How do you find boxes? It is hard • Decomposition is not unique (Non-Orthogonal) • Our Approach • Greedy Strategy for View Dependent • K-d subdivision for View Independent [Matusik et al 04]

22. Non-Linear Approximation View Dependent (specular) View Independent (diffuse) Original Original Approximated Approximated

23. p Overview of our method • Pre-compute Transport at a sparse set of sample view directions • Backwards Photon Tracing • Approximate Transport • Non-Linear Parametric Representation • Render • Fast Lighting-Transport Product • View Point Interpolation

24. RenderingBox Kernels Lighting Approximated Transport Exit Radiance (outgoing color) Summed Area Table Lookup

25. ? p Rendering Novel Views • Only Pre-computed Transport Functions for sparse set of outgoing directions • Interpolate Outgoing Radiance ?

26. ? p Interpolate Parameters • Interpolate Box parameters • Position • Size • Weight • Drawbacks: Visibility • But is at least plausible • Shadowing is View Independent • Does not need to be interpolated

27. Interpolating Views Example Resulting Color Interpolate Parameters Our Approach Interpolate Radiance Values Standard Linear Fading

28. ? p p Summary • Compute Transport Matrix T at sparse set of sample view directions • Factor T into view dependent and view independent parts • Approximate T using Non-Linear Parametric Representation • Render by interpolating parameters from closest precomputed Transport Vectors

29. Video

30. Contribution • Non-Linear Representation • View Point Interpolation Technique • All-Frequency Relighting From Non-Fixed Viewpoints with Arbitrary Reflectance and Transport Effects

31. Future Work • Other Non-Linear Approximations • Formal Box Decomposition Method • Compression of Transport Vectors • Hardware Acceleration

32. Thank You