Inverse Volume Rendering with Material Dictionaries

1. Inverse Volume Rendering with Material Dictionaries Ioannis Gkioulekas1 Kavita Bala2 Shuang Zhao2 Anat Levin3 Todd Zickler1 1Harvard 2Cornell 3Weizmann

2. Translucency is everywhere skin food architecture jewelry

3. Subsurface scattering • outgoing direction • incident direction • isotropic • extinction coefficient σt(λ) • scattering coefficient σs(λ) • radiative transfer equation phase function p(θ, λ) • Chandrasekhar 1960, Ishimaru 1978

4. Parameters are important extinction coefficient σt(λ) scattering coefficient σs(λ) • phase function p(θ, λ)

5. Parameters are non-trivial ≈ ≠

6. Light transport is complicated • rendering: • Monte-Carlo techniques • accurate (exact at convergence) • efficient (relatively) • rendering • (σtσs p(θ)) • Veach 1997

7. Light transport is complicated • inverse rendering: • fancy term for image-based parameter estimation • no standard way to solve it • rendering • (σtσs p(θ)) • inverse rendering • Bal 2009

8. Inverse rendering approximations • single-scattering approximation [Narashimhanet al. 2006] [Mukaigawaet al. 2010] • thin materials, diluted liquids • only approximate isolation • (σtσs p(θ)) • inverse rendering

9. Inverse rendering approximations • diffusion approximation • … • … • … • … [Jensen et al. 2001] [Papas et al. 2013] • optically thick materials • parameter ambiguity • (σtσs p(θ)) • inverse rendering • Wyman et al. 1989, Jensen et al. 2001, Papas et al. 2013

10. Inverse rendering approximations • parameterization of phase function [Chen et al. 2006] • Henyey-Greenstein lobes [Donner et al. 2008] [Fuchs et al. 2007] [Goesele et al. 2004] [Gu et al. 2008] [Hawkins et al. 2005] • single-parameter family: [Holroyd et al. 2011] [McCormick et al. 1981] • not general enough [Pine et al. 1990] [Prahl et al. 1993] [Gkioulekas et al. 2013] [Wang et al. 2008]

11. Our contributions • dictionary-based material representation • operator-theoretic framework for inverse volume rendering without approximations • physical setup implementation and accurate measurement of broader classes of materials

12. Basic idea • appearance matching • collect images μ1 μ2 μ3 μ6 μ4 μ5 μ9 μ7 μ8 • find material that matches them • m = (σtσs p(θ)) min Σ ǁ μs - render(m, scenes) ǁ2 m

13. Dictionary parameterization • dictionary of phase functions • p5 • p1 • p2 • p3 • p7 • p4 • p8 • p9 • p10 • p11 • p6 • D = {p1, p2, …, pQ} • express arbitrary phase function in terms of dictionary p • p = Σ wqpq • similarly for all parameters

14. Dictionary parameterization • dictionary of materials • D = {m1, m2, …, mQ} • express arbitrary material in terms of dictionary • m = Σ wqmq • similarly for all parameters

15. Basic idea • appearance matching • collect images μ1 μ2 μ3 μ6 μ4 μ5 μ9 μ7 μ8 • find material that matches them • m = (D, w) • m = (σtσs p(θ)) , scenes) ǁ2 m min Σ ǁ μs - render( w m w

16. Temporally resolved light transport propagation step τ = c Δt τ τ τ τ propagation probabilities: • extinction coefficient σt • continue straight: • scattering coefficient σs • π1 = 1 - τσt • phase function p(θ) π2 =τ (σt - σs) absorption event: volume event π3 =τσs(p(θ)) scattering event:

17. Temporally resolved light transport linear algebra of single propagation step τ = c Δt Lt+Δt= K Lt Lt Lt+Δt= (π1I + π3 P) Tτ single-step matrix K • straight: Lt+Δt,str= Tτ Lt • π1 = 1 - τσt • function of material: m = (σtσs p(θ)) π2 =τ (σt - σs) Lt+Δt,abs= 0 absorption: lightfield L Lt+Δt,sca= P Tτ Lt scattering: π3 =τσs

18. Temporally resolved light transport Li linear algebra of multiple L1 = K Li propagation steps τ = c Δt L2 = K2 Li Lt+Δt= K Lt L3= K3 Li single-step matrix K L4= K4 Li L5= K5 Li L= Σ KnLi = (I - K)-1 Li L6= K6 Li lightfield L t = 9 Δt t = 8 Δt t = 5 Δt t = 4 Δt t = 2 Δt t = 0 t = 6 Δt t = 3 Δt t = Δt t = 7 Δt rendering matrix R L7= K7 Li L8= K8 Li

19. Dictionary parameterization • dictionary of materials • D = {m1, m2, …, mQ} • express arbitrary material in terms of dictionary • m = Σ wqmq • single-step matrix K becomes • K = Σ wqKq • rendering matrix R becomes R = (I - Σ wqKq)-1

20. Basic idea • appearance matching • collect images μ1 μ2 μ3 μ6 μ4 μ5 μ9 μ7 μ8 • find material that matches them • m = (D, w) , scenes) ǁ2 min Σ ǁ μs - Ss (I - Σ wqKq)-1Li,s) ǁ2 min Σ ǁ μs - render( w w

21. Optimization • appearance matching loss function E(w) = Σ ǁ μs - Ss (I - Σ wqKq)-1Li,s) ǁ2 • differentiable! = 2wqΣ(Ss(I - Σ wqKq)-1Li,s -μs) Ss(I - Σ wqKq)-1Kq (I - Σ wqKq)-1Li,s • after some simplification = 2wqΣ(Ss R(w) Li,s -μs)SsR(w)Kq R(w) Li,s • use your favorite gradient descent variant to do inverse rendering

22. The world is continuous linear algebra of temporally resolved light transport Lt+Δt= K Lt single-step matrix K L= R Li = (I - K)-1 Li rendering matrix R lightfield L • light transport can be described using matrix-vector products (continuous function)

23. The world is continuous operator theory of temporally resolved light transport Lt+Δt= K Lt single-step operator K L= R Li = (I - K)-1 Li rendering operator R lightfield L • matrix-vector products become (Monte-Carlo) rendering (continuous function) • Arvo 1995

24. Optimization (operator version) operator version of gradient is identical = 2wqΣ(Ss R(w) Li,s -μs)SsR(w)Kq R(w) Li,s • three cascaded Monte-Carlo operations • noisy estimates use stochastic gradient descent R(w) R(w) Kq

25. Basic idea • appearance matching • collect images μ1 μ2 μ3 μ6 μ4 μ5 μ9 μ7 μ8 • find material that matches them • m = (D, w) min Σ ǁ μs - Ss (I - Σ wqKq)-1Li,s) ǁ2 w

26. Acquisition setup front lighting material sample backlighting hyperspectral camera

27. Acquisition setup front lighting material sample backlighting hyperspectral camera top rotation stage bottom rotation stage

28. Acquisition setup material sample frontlighting front lighting material sample backlighting hyperspectral camera top rotation stage top rotation stage hyperspectral camera bottom rotation stage

29. Acquisition setup MEMS light switch blue (480 nm) laser green (535 nm) laser red (635 nm) laser RGB combiner light generation

30. Acquisition setup • Some acquisition statistics: • 3 lighting angles (for both front- and back-lighting) • 3 viewing angles • LDR images at 18 exposures • 3 wavelengths • total 54 HDR measurements per material • 75 minutes capture per material

31. Validation materials reference nanodispersions • particles of known molecular type dispersed in liquid media • very precise size distributions (NIST Traceable Standards) • ground-truth materials: parameters predicted exactly by Mie theory polystyrene monodispersions aluminum oxide polydispersions • Frisvad et al. 2007

32. Validation materials comparison of predicted and measured parameters polystyrene 1 polystyrene 2 polystyrene 3 aluminum oxide all parameters estimated within 4% error

33. Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles polystyrene 1 images under unseen geometries predicted within 5% RMS error

34. Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles polystyrene 2 images under unseen geometries predicted within 5% RMS error

35. Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles polystyrene 3 images under unseen geometries predicted within 5% RMS error

36. Validation materials comparison of captured and rendered images in novel geometries captured rendered rendered with HG profiles aluminum oxide images under unseen geometries predicted within 5% RMS error

37. Measured materials hand cream curacao • highly scattering liquids olive oil shampoo robitussin mixed soap whole milk • highly absorbing liquids • solids milk soap wine liquid clay mustard coffee reduced milk

38. Measured materials whole milk reduced milk shampoo hand cream mustard liquid clay milk soap mixed soap glycerine soap robitussin curacao wine coffee olive oil

39. Measured materials

40. Measured materials

41. Measured materials

42. Discussion • more interesting materials: heterogeneous volumes, polarization-dependent behavior, birefringent and fluorescing materials • use of our optimization framework with other setups: alternative lighting (basis, adaptive, high-frequency), geometries, or imaging (transient imaging) • simpler optimization and setup: trade-offs between accuracy, generality, mobility, and usability

43. Take-home messages • dictionary-based material representation • operator-theoretic framework for inverse volume rendering without approximations • physical setup implementation and accurate measurement of broader classes of materials

44. Acknowledgments • Shuang Zhao • KavitaBala • Todd Zickler • Anat Levin

45. Thank you for your attention