View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations

1. View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations Paul Green1 Jan Kautz1 Wojciech Matusik2 Frédo Durand1 MIT CSAIL1 MERL2 ACM Symposium in Interactive 3D Graphics 2006

2. Interactive 6D Relighting Geometry & Viewpoint Reflectance All-Frequency Lighting Rendered Frame

3. Goal: 6D Relighting • High-quality view-dependent effects • Sharp highlights • Spatially varying BRDFs • Allow rendering w/ large environment maps (e.g. 6x256x256) • Without paying a prohibitive data storage price

4. Applications • Games • Architectural Visualization

5. Outline • Background / Previous Work • PRT • Nonlinear Approximation • Our Representation • Rendering • Fitting • Results • Conclusion

6. Shadowing Inter-reflection Incident Radiance Precomputed Light Transport Transport function maps distant light to incident light Can Include BRDF if outgoing direction ωo is fixed Distant Radiance Exit Radiance Courtesy of Sloan et al. 2003

7. Courtesy of Sloan et al. 2003 Light Transport • RadianceLoat pointpalong directionisweighted sum of distant radianceLi Outgoing Radiance Transport Vector Distant Radiance (Environment Map)

8. Example It’s a Dot Product Between Lighting and Transport Vectors!! Transport Function Environment Map Exit Radiance (outgoing color) BRDF Weighted Incident Radiance

9. Light Transport • Data Size Problem • Many GB’s of data • Rendering is slow • 6x64x64 cubemap ~24,000 mults/vert • Can reduce size in different basis: • Spherical Harmonics • Wavelets • Zonal Harmonics

10. PRT with Spherical Harmonics • Precomputed Radiance Transfer [Sloan et al 02,03] • Low Order Spherical Harmonics • Soft Shadows and Low Frequency Lighting • Not Suitable For Highly Glossy Materials • Not Practical For High Frequency Lighting Image from slides by Ng et al.2003

11. Nonlinear Wavelet PRT • Nonlinear Wavelet Lighting Approximation [Ng et al 03] • Haar Wavelets • Nonlinear Approximation • All Frequency Lighting • Fixed View For Arbitrary BRDFs Image from slides by Ng et al.2003

12. Separable PRT • Factor BRDF into product of view-only and light-only functions [Liu et al 04, Wang et al 04] • Nonlinear Wavelet Approximation [Ng et al 03] • Need factorization per BRDF • Very specular materials still require many coefficients Liu et al 04

13. Our Goal: 6D Relighting • High quality view-dependent effects • Representation of transport that enables • Spatially varying BRDFs • Arbitrary highlight scale • compact storage • High-res environment maps (e.g. 6x256x256) • Sparse view samplingRequires high-quality interpolation • Over view directions • Over mesh triangles

14. Outline • Background / Previous Work • PRT • Nonlinear Approximation • Our Representation • Rendering • Fitting • Results • Conclusion

15. Factoring Transport Per view View-dependent ? Nonlinear Gaussian Function Approximation Represent with SH or Wavelets View-independent (diffuse)

16. - mean - std. deviation Our Nonlinear Representation Sum of N isotropic Gaussians

17. Nonlinear: 8 Largest Coefficients SSE = 25.1 Previous work: Nonlinear Wavelet • Nonlinear: Approximating basis set depends on input • Truncate small coefficients • Effect of coefficients is still linear Linear: First 8 Coefficients SSE = 140.2

18. Nonlinear: 8 Largest Coefficients Nonlinear sum of 2 Gaussians SSE = 5.61 SSE = 25.1 Our solution: Even More Nonlinear • We don’t start from linear basis • No Truncation of coefficients • Nonlinear parameter estimation • Parameters have nonlinear effects

19. Haar 70 terms original N = 2 N = 1 Advantages of sum of Gaussians • Arbitrary freq bandwidth • Accurate approx w/small storage • Good interpolation • Good visual quality

20. p Examples

21. Rendering Lighting Approximated Transport ?

22. Gaussian Pyramid Pre-convolve environment with Gaussians of varying sizes Only done Once Can start with large cubemap e.g. 6x256x256 Larger σ

23. Rendering Lighting Approximated Transport Exit Radiance (outgoing color) Tri-linear lookup in Gaussian Pyramid

24. ? p Rendering Novel Views • Precomputed Transport Functions for sparse set of outgoing directions • Naïve solution: (Gouraud Shading) Interpolate Outgoing Radiance Cross-fading artifacts

25. ? p Better: Interpolate Parameters • Interpolate Gaussian parameters • Mean, Std. Dev, Weights • Analogous to Phong vs Gouraud shading t=0 t=0.5 t=1

26. Per-Pixel Interpolation

27. Interpolation Drawbacks • Visibility may not interpolate correctly • But is usually plausible • Correspondences • Makes fitting more difficult View-dependent View-independent

28. p Data Fitting • Precompute Raw Transport Data • Solve large scale nonlinear optimization problem • For each vertex • For each view • Fit Gaussians to transport data

29. - N = 1 p Nonlinear Optimization original N = 2 Objective has terms for • Fitting the Data • Regularization • Angular smoothness and correspondences • Spatial smoothness and correspondences

30. Error Plots

31. Results • 6x256x256 Environment Maps • Single Gaussian • 2.8 GHz P4 • 1GB RAM • Nvidia 6800 Ultra • Software screen capture

32. original N = 2 Haar Contributions • New parametric representation of PLT • Compact Storage • High Quality • Interpolation of parameters • Sparse View Sampling • Efficient Rendering • Spatially varying BRDFs

33. Questions?