Inverse Volume Rendering with Material Dictionaries

1. Inverse Volume Rendering with Material Dictionaries Ioannis Gkioulekas1 Kavita Bala2 Shuang Zhao2 Todd Zickler1 Anat Levin3 1Harvard 2Cornell 3Weizmann

2. Most materials are translucent skin food jewelry architecture • Photo credit: Bei Xiao, Ted Adelson

3. We know how to render them • Monte-Carlo rendering ? • material parameters rendered image • Veach 1997, Dutré et al. 2006

4. We show how to measure them • inverse rendering • material parameters rendered image captured photograph

5. Our contributions 1. exact inverse volume rendering • with arbitrary phase functions! 2. validation with calibration materials • material • known parameters • thick • thin 3. database of broad range of materials • non-dilutable • solids

6. Why is inverse rendering so hard? random walk of photons inside volume • radiativetransfer • volume light transporthas very complex dependence material parameters material sample • thick • thin • non-dilutable • solids

7. Light transport approximations single-bounce random walk random walk of photons inside volume • Previous approach: single-scattering Narasimhan et al. 2006   • thick • thin   • non-dilutable • solids

8. Light transport approximations isotropic distribution of photons random walk of photons inside volume • Previous approach: diffusion • … • … • … • … Papas et al. 2013 Jensen et al. 2001 • parameter ambiguity   • material 1 • thick • thin ≈ ≠   • non-dilutable • solids • material 2

9. Inverse rendering without approximations exact inversion of random walk random walk of photons inside volume   • thick • thin   • non-dilutable • solids

10. Our approach • appearance matching • i. material representation • ii. operator-theoretic analysis • iii. stochastic optimization

11. Background random walk of photons inside medium θ • extinction coefficient σt • m = (σtσs p(θ)) • scattering coefficient σs phase function p(θ)

12. Phase function parameterization • Previous approach: single-parameter families • Henyey-Greenstein lobes Chen et al. 2006 Donner et al. 2008 Fuchs et al. 2007 Goesele et al. 2004 Gu et al. 2008 Hawkins et al. 2005 • not general enough Holroyd et al. 2011 Jensen et al. 2001 Gkioulekas et al. 2013 McCormick et al. 1981 Narasimhanet al. 2006 Papas et al. 2013 Pine et al. 1990 Prahl et al. 1993 Wang et al. 2008

13. Dictionary parameterization • tent phase functions • dictionary of • phase functions • materials • D = {m1, m2, …, mQ} • D = {p1, p2, …, pQ} • p11 • p10 • p9 • p8 • p7 • p3 • p5 • p4 • p2 • p1 • p6 • D • arbitrary • phase functions • materials • π5 • π4 • π3 • π7 • π6 • π8 • π9 • π2 • π10 • m = Σqπqmq • p = Σqπqpq • π1 • π11 p • similarly for σt and σs • σt = Σqπqσt,q • σs = Σqπqσs,q

14. Our approach • appearance matching • i. material representation • m = Σqπqmq • ii. operator-theoretic analysis • iii. stochastic optimization

15. Operator-theoretic analysis random walk of photons inside medium • discretized random walk paths • propagation step τ τ τ τ τ • m = (σtσs p(θ))

16. Operator-theoretic analysis radiance at all medium points and directions • discretized random walk paths • propagation step τ Ln+1(x, θ)= Ln(x, θ) K radiance after n steps radiance after n+1 steps • total radiance L= ΣnLn = (I - K)-1Linput • rendering operator R L(x, θ)= R Linput(x, θ) L(x, θ) dictionary representation: • m = (σtσs p(θ)) • m = Σq πqmq • K(π) = Σq πqKq R(π)= (I - Σ q πqKq)-1

17. Our approach • appearance matching • i. material representation • m = Σqπqmq • ii. operator-theoretic analysis R(π)= (I - Σ q πqKq)-1 • iii. stochastic optimization

18. Stochastic optimization • appearance matching min ǁ photo - render(π) ǁ2 π • analytic operator expression for gradient! = render(π) · single-stepq · render(π) Kq R(π) R(π) • gradient descent optimization for inverse rendering

19. Stochastic optimization • exact gradient descent • for k = 1, …, N, N = a few hundreds * • πk = πk -1 - ak • several CPU hours = • end • intractable exact

20. Stochastic optimization • Monte-Carlo rendering to compute 106 samples 102 samples 104 samples • noisy + fast • accurate + slow

21. Stochastic optimization • exact gradient descent • for k = 1, …, N, • for k = 1, …, N, N = a few hundreds * • πk = πk -1 - ak • πk = πk -1 - ak • several CPU hours = • end • end • intractable • stochastic gradient descent exact noisy N = a few hundreds * • few CPU seconds = • solvable

22. Theory wrap-up • appearance matching min ǁ photo - render(π) ǁ2 π • i. material representation • m = Σqπqmq • ii. operator-theoretic analysis R(π)= (I - Σ q πqKq)-1 noisy • iii. stochastic optimization

23. Our contributions 1. exact inverse volume rendering • with arbitrary phase functions! 2. validation with calibration materials • material • known parameters • thick • thin 3. database of broad range of materials • non-dilutable • solids

24. Measurements • appearance matching min ǁ photo - render(π) ǁ2 π • multiple lighting multiple viewpoints

25. Acquisition setup material sample frontlighting camera backlighting

26. material sample frontlighting Acquisition setup backlighting material sample frontlighting camera backlighting bottom rotation stage camera top rotation stage top rotation stage bottom rotation stage

27. Validation • calibration materials • medium material Mie theory • particle material % size • known parameters very precise dispersions (NIST Traceable Standards) aluminum oxide polydispersions polystyrene monodispersions • Frisvad et al. 2007

28. Parameter accuracy • comparison of ground-truth and measured parameters p(θ) polystyrene 1 polystyrene 2 polystyrene 3 aluminum oxide θ all parameters estimated within 4% error ground-truth measured • -π • 0 • π Henyey-Greenstein fit

29. Matching novel measurements comparison of captured and rendered images rendered captured rendered with HG profiles polystyrene 3 images under unseen geometries predicted within 5% RMS error ground-truth measured Henyey-Greenstein fit

30. Our contributions 1. exact inverse volume rendering • with arbitrary phase functions! 2. validation with calibration materials • material • known parameters • thick • thin 3. database of broad range of materials • non-dilutable • solids

31. Measured materials hand cream olive oil curacao shampoo robitussin mixed soap whole milk milk soap wine liquid clay mustard reduced milk coffee • thick • thin • non-dilutable • solids

32. Measured phase functions whole milk reduced milk shampoo hand cream mustard liquid clay milk soap mixed soap glycerine soap robitussin p(θ) θ • -π • 0 • π curacao wine olive oil coffee measured Henyey-Greenstein fit

33. Synthetic images mixed soap curacao glycerine soap olive oil whole milk rendered image

34. Synthetic images chromaticity

35. Synthetic images mixed soap curacao glycerine soap olive oil whole milk rendered image

36. Effect of phase function measured phase function Henyey-Greenstein fit chromaticity rendered image p(θ) mixed soap θ measured • -π • 0 • π Henyey-Greenstein fit

37. Discussion • more interesting materials: more general solids, heterogeneous volumes, fluorescing materials • other setups: alternative lighting (basis, adaptive, high-frequency), geometries, or imaging (transient imaging) • faster capture and convergence: trade-offs between accuracy, generality, mobility, and usability

38. Take-home messages 1. exact inverse volume rendering • with arbitrary phase functions! 2. validation with calibration materials • material • known parameters • thick • thin 3. database of broad range of materials • non-dilutable • solids

39. Acknowledgements • Henry Sarkas (Nanophase) • Wenzel Jakob (Mitsuba) • Funding: • National Science Foundation • European Research Council • BinationalScience Foundation • Feinberg Foundation • Intel • Amazon • Database of measured materials: http://tinyurl.com/sa2013-inverse

40. Error surface • appearance matching min ǁ photo - render(π) ǁ2 π

41. blue (480 nm) laser MEMS light switch Light generation red (635 nm) laser green (535 nm) laser RGB combiner