All-Frequency Shadows Using Non-linear Wavelet Lighting Approximation

1. All-Frequency Shadows Using Non-linear Wavelet Lighting Approximation

2. Lighting Design • From Frank Gehry Architecture, Ragheb ed. 2001

3. Lighting Design • From Frank Gehry Architecture, Ragheb ed. 2001

4. Existing Fast Shadow Techniques • We know how to render very hard and very soft shadows Sen, Cammarano, Hanrahan, 2003 Sloan, Kautz, Snyder 2002 Shadows from smooth lighting (precomputed radiance transfer) Shadows from point-lights (shadow maps, volumes)

5. All-Frequency Lighting Teapot in Grace Cathedral

6. Soft Lighting Teapot in Grace Cathedral

7. All-Frequency Lighting Teapot in Grace Cathedral

8. Soft Lighting Teapot in Grace Cathedral

9. All-Frequency Lighting Teapot in Grace Cathedral

10. Overview • Matrix formulation of relighting • Pre-computing the light-transport matrix • Compressing the matrix multiplication • Results • Demonstration • Error analysis (Comparison with PRT) • Future directions • Limitations • Areas for research

11. Overview • Matrix formulation of relighting • Pre-computing the light-transport matrix • Compressing the matrix multiplication • Results • Demonstration • Error analysis (Comparison with PRT) • Future directions • Limitations • Areas for research

12. Relighting as Matrix-Vector Multiply

13. Relighting as Matrix-Vector Multiply • Input Lighting(Cubemap Vector) • Output Image(Pixel Vector) • TransportMatrix

14. Ray-Tracing Matrix Columns

15. Ray-Tracing Matrix Columns

16. Light-Transport Matrix Rows

17. Light-Transport Matrix Rows

18. Light-Transport Matrix Rows

19. Rasterizing Matrix Rows • Pre-computing rows • Rasterize visibility hemicubes with graphics hardware • Read back pixels and weight by reflection function

20. Overview • Matrix formulation of relighting • Pre-computing the light-transport matrix • Compressing the matrix multiplication • Results • Demonstration • Error analysis (Comparison with PRT) • Future directions • Limitations • Areas for research

21. Overview • Matrix formulation of relighting • Pre-computing the light-transport matrix • Compressing the matrix multiplication • Results • Demonstration • Error analysis (Comparison with PRT) • Future directions • Limitations • Areas for research

22. Matrix Multiplication is Enormous • Dimension • 512 x 512 pixel images • 6 x 64 x 64 cubemap environments • Full matrix-vector multiplication is intractable • On the order of 1010 operations per frame

23. Sparse Matrix-Vector Multiplication • Choose data representations with mostly zeroes • Vector: Use non-linear wavelet approximation on lighting • Matrix: Wavelet-encode transport rows

24. Non-linear Wavelet Light Approximation • Wavelet Transform

25. Non-linear Wavelet Light Approximation • Non-linearApproximation • Retain 0.1% – 1% terms

26. Why Non-linear Approximation? • Linear • Use a fixed set of approximating functions • Precomputed radiance transfer uses 25 - 100 of the lowest frequency spherical harmonics • Non-linear • Use a dynamic set of approximating functions (depends on each frame’s lighting) • In our case: choose 10’s - 100’s from a basis of 24,576 wavelets • Idea: Compress lighting by considering input data

27. Why Wavelets? • Wavelets provide dual space / frequency locality • Large wavelets capture low frequency, area lighting • Small wavelets capture high frequency, compact features • In contrast • Spherical harmonics • Perform poorly on compact lights • Pixel basis • Perform poorly on large area lights

28. Choosing Non-linear Coefficients • Three methods of prioritizing • Magnitude of wavelet coefficient • Optimal for approximating lighting • Transport-weighted • Biases lights that generate bright images • Area-weighted • Biases large lights

29. Sparse Matrix-Vector Multiplication • Choose data representations with mostly zeroes • Vector: Use non-linear wavelet approximation on lighting • Matrix: Wavelet-encode transport rows • Choose data representations with mostly zeroes • Vector: Use non-linear wavelet approximation on lighting • Matrix: Wavelet-encode transport rows

30. Matrix Row Wavelet Encoding

31. Matrix Row Wavelet Encoding Extract Row

32. Matrix Row Wavelet Encoding Wavelet Transform

33. Matrix Row Wavelet Encoding Wavelet Transform

34. Matrix Row Wavelet Encoding Wavelet Transform

35. Matrix Row Wavelet Encoding Wavelet Transform

36. Matrix Row Wavelet Encoding 0 0 0 ’ ’ Store Back in Matrix

37. Matrix Row Wavelet Encoding 0 0 0 ’ ’ Only 3% – 30% are non-zero

38. Total Compression • Lighting vector compression • Highly lossy • Compress to 0.1% – 1% • Matrix compression • Essentially lossless encoding • Represent with 3% – 30% non-zero terms • Total compression in sparse matrix-vector multiply • 3 – 4 orders of magnitude less work than full multiplication

39. Overall Relighting Algorithm • Pre-compute (per scene) • Compute matrix in pixel basis • Wavelet transform rows • Quantize, store • Interactive Relighting (each frame) • Wavelet transform lighting • Prioritize and retain N wavelet coefficients • Perform sparse-matrix vector multiplication

40. Overview • Matrix formulation of relighting • Pre-computing the light-transport matrix • Compressing the matrix multiplication • Results • Demonstration • Error analysis (Comparison with PRT) • Future directions • Limitations • Areas for research

41. Overview • Matrix formulation of relighting • Pre-computing the light-transport matrix • Compressing the matrix multiplication • Results • Demonstration • Error analysis (Comparison with PRT) • Future directions • Limitations • Areas for research

42. Demo

43. Error Analysis • Compare to linear spherical harmonic approximation • [Sloan, Kautz, Snyder, 2002] • Measure approximation quality in two ways: • Error in lighting • Error in output image pixels • Main point (for detailed natural lighting) • Non-linear wavelets converge exponentially faster than linear harmonics

44. Error in Lighting: St Peter’s Basilica • Sph. Harmonics • Non-linear Wavelets • Relative L2 Error (%) • Approximation Terms

45. Error in Output Image: Plant Scene • Sph. Harmonics • Non-linear Wavelets • Relative L2 Error (%) • Approximation Terms

46. Output Image Comparison • 25 • 200 • 2,000 • 20,000 • Top: Linear Spherical Harmonic ApproximationBottom: Non-linear Wavelet Approximation

47. Summary • A viable representation for all-frequency lighting • Sparse non-linear wavelet approximation • 100 – 1000 times information reduction • Efficient relighting from detailed environment maps

48. Future Improvements • Matrix compression • Transport matrices are very large (over 1 GB) • We only compress matrix rows • Can also compress columns (images) • Non-linear coefficient selection • Area-weighted priority works best • Develop theory for how to approximate input vector for minimum error in output vector

49. Signal Processing Taxonomy of Lighting Effects • “All-frequency” vs “Exclusively low frequency”

50. Future All-Frequency Lighting Research • Realistic materials with changing view • Local lighting effects • Gehry’s E.M.P. Museum