Interactive Editing and Modeling of Bidirectional Texture Functions

1. Interactive Editing and Modeling of Bidirectional Texture Functions Frédo Durand MIT – CSAIL Jan Kautz University College London Solomon Boulos University of Utah

2. Introduction • Bidirectional Texture Functions (BTFs) • A representation for complex materials [Dana99] • Collection of textures under many different viewand light directions • 6D data structure: 2D spatial, 2D view, 2D light view: top light: top view: top light: 50º view: 45º light: 50º view: 50º light: 60º

3. Introduction • Bidirectional Texture Functions • Usually acquired from real materials, e.g. [Sattler03] • Include real-world effects (scattering, occlusions, transparencies, parallax, …) • Very realistic Lamp Camera Sample(on robotic arm)

4. Introduction • Acquisition of Bidirectional Texture Functions • Is very expensive (time, setup, …) • University of Bonn BTFs have: • 81 view x 81 light images (6561 images) • Takes several hours to capture • Only 10 publicly available

5. Introduction • Acquisition of Bidirectional Texture Functions • Is very expensive (time, setup, …) • University of Bonn BTFs have: • 81 view x 81 light images (6561 images) • Takes several hours to capture • Only 10 publicly available Want better return on investment:  Editing of BTFs Allows to create variations from single BTF Create BTF from flat texture

6. Introduction • How to edit BTFs?

7. Introduction • How to edit BTFs? • Everything is baked in • No explicit knowledge of • BRDFs • Geometry • Transparencies , etc. • Extraction of these properties is • Difficult • How to support editing of high-level properties

8. Related Work • Most work focuses on • BTF Acquisition [Dana99,Sattler03,Koudelka03,…] • BTF Rendering/Compression [Sattler03, Meseth04,Vasilescu04,…] • Very little on BTF editing • Pasting BTFs onto surfaces [Zhou05] • Self-similarity-based editing [Dong05] • Structural editing of BTFs [Mueller07] • BRDF editing [Lawrence06, Colbert06]

9. Our BTF Editing Approach • Simple operators • Can be applied selectively (with selections) • Enables powerful editing: “BTFShop” VIDEO

10. Our BTF Editing Approach • Philosophy • Simple operators → change material appearance • Rely on simple aspects of material perception: • Contrast ↔ roughness [Koenderink02] • Histograms ↔ roughness [Ginneken99, Leung97] • Sharpness ↔ subsurface scattering [Fleming04] → Enables simple operators to be effective

11. Our BTF Editing Operators Operators • E.g., color manipulation, blur/sharpen • Work on certain dimensions • The simpler the better • Use raw data, avoid big inverse problems • Exploit shape information, when appropriate • Constrained to user selections • E.g., highlight pixels, directional range

12. Overview • Shading • Shadowing • Geometry • Global Effects Due to time: only highlight one example each

13. Overview • Shading • Shadowing • Geometry • Global Effects • Low-level operators • Blur / sharpen • Color manipulation • Give direct control of simple material properties • Shading & roughness

14. lower-frequency higher-frequency Shading – Changing Specularity • Goal: • Specularity relates to frequencies in light-space [Ramamoorthi01]: blur/sharpen in light-space Light-dependencefor given view and location Light-dependencefor given view and location

15. Original Blur in Light-Space Reference Shading – Changing SpecularityResults – Part 1 (BTF computed from synthetic micro-geometry) (BTF computed from synthetic micro-geometry)

16. Blur in Light-Space Original Sharpen in Light-Space Shading – Changing SpecularityResults – Part 2 (Measured BTF)

17. Shading – Changing Roughness • Goal: modify roughness ?

18. Output Intensity Input Intensity Shading – Changing Roughness • Observation: • Distribution of light vs. dark pixels relates to roughness [Pont and Koenderink 02] • Distribution can be changed with curve tool • User-controlledcurve • Remaps input to output intensities

19. remappingcurve BTFShop Reference (BTF: ) synthetic micro-geometry Original (BTF: ) synthetic micro-geometry

20. Overview • Shading • Shadowing • Geometry • Global Effects • More complex editing operations • Requiring selections for effective modification

21. Shadow Removal • Goal: • Challenges: • How to find shadow areas? • How to fill them in automatically?

22. Shadow Removal –Finding Shadow Areas • User-assisted selection • Fully-lit, frontal texture slice: always shadow-free • Check ratio of: fully-lit frontal slice / each of light-varying frontal slices • If (scaled) ratio < user-threshold: pixel in shadow / = fully-lit, frontal side-lit, frontal ratio thresholdapplied

23. fully-lit, frontal fully-lit, frontal side-lit, frontal side-lit, frontal ratio ratio thresholded thresholded Shadow Removal –Finding Shadow Areas / = / =

24. Shadow Removal –Finding Shadow Areas Found Shadow Areas

25. Shadow Removal –Automatically Filling in Shadows Two-stage process • Fill in selected areas with data from fully-lit frontal texture (never contains shadows) • Adjust average brightness and saturation to match surrounding area

26. Shadow Removal –Automatically Filling in Shadows Original

27. Shadow Removal –Automatically Filling in Shadows Selected Shadow Areas

28. Shadow Removal –Automatically Filling in Shadows Shadow Areas Filled In From Fully-Lit, Frontal Texture

29. Shadow Removal –Automatically Filling in Shadows Brightness/Saturation Adjusted

30. Label Shadow Removal – Results Shading is preserved Original Shadows Removed

31. Overview • Shading • Shadowing • Geometry • Global Effects • Now: • Editing operator requiring inferred information

32. Rendering (x1,y1) (x2,y2) Geometry – Parallax • Parallax is common in complex materials • Geometric point → different (x,y)

33. Geometry – Parallax • Shows up in BTF slices: • Consequences: • Spatial editing such as painting needs to take it into account • Change of geometric structure → warp in texture space Frontal Slice Oblique Slice Frontal Slice Oblique Slice Original Height Increased Height

34. Parallax – Spatial Editing Extract Heightfield (Shape from Shadows) Unwarp Texture Slices (Removes Parallax, Aligns Texture Slices) Spatial Editing Warp Texture Slices (Reintroduce Parallax)

35. Parallax – Spatial Editing Result No Correction Parallax Correction

36. Geometric Modification • Parallax Warp can also be used to impose different geometric structure Heightfield

37. Local Geometric Appearance • Parallax Warp • Does not modify underlying shading • Bump mapping [Blinn78]: modify local view and light direction + = Diffuse Heightfield Bump Mapped

38. Local Geometric Appearance • Use same idea for BTFs • Rotate lookup directions w.r.t. heightfield • Lookup values with rotated directions • Store values again as BTF

39. Bump Mapped BTF Bump Mapped + Warped BTF Local Geometric Appearance Original BTF Heightfield

40. Overview • Shading • Shadowing • Geometry • Global Effects

41. BTFShop Global Effects • Asperity Scattering [Koenderink03] • “Glow” at grazing angles • Simulate using angular selection and curve tool No Fuzz (BTF: synth. micro-geo.) Fuzz – Reference (BTF: synth. micro-geo.)

42. Use to modifyBTFs Overview • Shading • Shadowing • Geometry • Global Effects • More tools / selections: see paper Edit Albedos Modify Shadows Edit Thickness Edit Subsur. Scat.

43. More Results VIDEO

44. Use to createBTFs Overview • Shading • Shadowing • Geometry • Global Effects • More tools / selections: see paper

45. Modeling of BTFs Single Texture BTF Created

46. Modeling of BTFs Single Texture Created BTF Reference BTF

47. Modeling of BTFs VIDEO

48. BTFShop – System • Multi-threaded, tile-based architecture with efficient out-of-core caching • Visual feedback in ~2s (touch 5% data) • Complete BTF (1.6GB data): ~minutes

49. Discussion • Approach works well for tweaking BTF appearance: • Shading, Shadowing, Geometry, and Global Effects • Limitations • Cannot turn bark into wool • Resulting BTFs not physically correct • Operators edit one effect at a time

50. Conclusions • Simple operators and selections → surprisingly effective • Mostly work on raw 6D data • Infer properties when necessary • BTFShop: out-of-core (6D dataset) • BTF • Modification • Creation from single textures