Orientation fields and 3D shape estimation

1. Roland W. Fleming Max Planck Institute for Biological Cybernetics Orientation fields and 3D shape estimation

2. Cues to 3D Shape specularities shading texture Conventional wisdom: different cues have different physical causes  must be processed differently by visual system (‘modules’)

3. Cues to 3D Shape specularities shading texture Goal: Find commonalities between cues.

4. Cues to 3D Shape

5. Cues to 3D Shape Zaidi and Li Fleming, Torralba, Adelson Zucker and colleagues Todd and colleagues Koenderink and van Doorn Malik and Rosenholtz Mingolla and Grossberg

6. Shape from Specularities Ideal mirrored surface • It is remarkable that we can recover 3D shape: • No motion • No stereo • No shading • No texture • image consists of nothing more than a distorted reflection of the world surrounding the object Fleming et al. (2004). JOV

7. Shape from Specularities As the object moves from scene to scene, the image changes dramatically. Yet, somehow we are able to recover the 3D shape.

8. Curvatures determine distortions highly curved

9. Curvatures determine distortions slightly curved Anisotropies in surface curvature lead to powerful distortions of the reflected world

10. Interpreting distorted reflections

11. Orientation fields Ground truth

12. 3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface

13. Beyond specularity Specular reflection Diffuse reflection

14. Differences betweendiffuse and specular reflection

15. Differences betweendiffuse and specular reflection

16. Differences betweendiffuse and specular reflection

17. Shiny Painted

18. Beyond specularity Specular reflection Diffuse reflection

19. Latent orientationstructure

20. Orientation fieldsin shading

21. Orientation fieldsin shading

22. Reflectance as Illumination a(f) = 1 / f  = 0  = 0.4  = 0.8  = 1.2  = 1.6  = 2.0  = 4.0  = 8.0

23. highly curved

24. slightly curved Anisotropies in surface curvature lead to anisotropies in the image.

25. Stability across changesin surface reflectance • A parametric space of glossy plastic materials (using Ward model) Specular Reflectance, s Diffuse Reflectance, d

26. Idea: Experiment 1 • Rationale: measure stability of 3D shape across changes in surface reflectance • Method: gauge figure task? • Problem: costly to do full depth reconstruction for many shapes and materials • Solution? Compare sparse gauge measurement? • Alternative task?: • locate depth extrema along given raster line (2D task)

27. Texture Anisotropic compression of texture depends on surface slant

28. Texture Anisotropic compression of texture depends on surface slant

29. Orientation fieldsin texture

30. Orientation fieldsin texture

31. Orientation fieldsin texture

32. Affine Transformation • Shear: • does affect first • derivatives • does NOT affect • second derivatives

33. Affine Transformation • Shear: • does affect first • derivatives • does NOT affect • second derivatives

34. Affine Transformation • Shear: • does affect first • derivatives • does NOT affect • second derivatives

35. Affine Transformation • Shear: • does affect first • derivatives • does NOT affect • second derivatives

36. Affine Transformation • Shear: • does affect first • derivatives • does NOT affect • second derivatives

37. Affine Transformation • Shear: • does affect first • derivatives • does NOT affect • second derivatives

38. Idea: Experiment 2 • Rationale: use orientation fields to predict misperceptions of 3D shape • Possible methods • Gauge figure task? • Matching task: • subject adjusts shear of a textured object until it appears to match the shaded version of the same object • Subject adjusts shear of one oject (shaded or textured) until it appears to match the ‘degree of shear’ of another object? Sounds too strange?

39. Illusory distortionsof shape Inspired by Todd & Thaler VSS 05

40. Illusory distortionsof shape Inspired by Todd & Thaler VSS 05

41. Idea: Experiment 3 • Rationale: use orientation fields to predict misperceptions of 3D shape • Possible methods • gauge figure task to reconstruct full 3D shape. • Again, this is costly, but perhaps a few shapes are enough • depth extrema task: locate depth extrema along raster line (this is what Todd and Thaler did). • Potentially we could predict the locus directly from the orientation field

42. Idea: Experiment 3 • Compare small and large changes in orientation field by using texture stretching along the line of sight • Advantage: same infringement of ‘isotropy assumption’, different change in apparent 3D shape Stretched 2:1 along line of sight Unstretched

43. Potential of Orientation Fields • Uses biologically plausible measurements Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997).

44. Potential of Orientation Fields • No need for visual system to estimate reflectance or illumination explicitly. • Classical shape from shading uses the reflectance map to estimate surface normals from image intensities • Reflectance map is usually unknown and ambiguous

45. Potential of Orientation Fields • Stable across albedo discontinuities. Breton and Zucker (1996), Huggins and Zucker (2001)

46. Potential of Orientation Fields • Handle improbable combinations of reflectance and illumination. normal shading ‘weird’ shading non-linear intensity transfer function

47. Link back toexperiment 1 ? • We could measures shape estimates with these types of stimuli as well. normal shading ‘weird’ shading non-linear intensity transfer function

48. Potential of Orientation Fields • May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts Ohad Ben-Shahar

49. 2 ( ) Converting between cues input image Todd & Oomes 2004 Latent shading

50. 2 ( ) Converting between cues input image Todd & Oomes 2004 Latent shading