Shape and Shading

1. Shape and Shading Koenderink & Van Dorn Chapter 72 of The Visual Neurosciences

2. Introduction • Observers are interested in • Geometry of objects • Material identification • Light field: the primary and secondary sources of radiation and how radiation pervades space • What we have: an image • What we are interested in: the scene

3. Introduction • Current topic deals with only the static, achromatic, monocular observer • Ignoring • Observer motion • Object motion • Color • Binocular • Material properties

4. Introduction • We want to know how object shape can be determined from monochrome pictures.

5. The light field • Light model: rays (particle model) • Radiators • Primary: luminous • Secondary • Reflecting • Scattering

6. The light field • Volume (ray) density of radiation: the length of all rays crossing a volume, divided by that volume. • Or the density of photons crossing that volume during a given time interval.

7. The light field • Net flux vector: • Number of photons crossing a unit area per unit time. • Calculated as proportional to cos(), where  is the angle between a surface patch normal and the beam direction. • Such a definition is appropriate only for small patches, of course.

8. The light field • The light passing thru any such circular patch defines a tube. • Such tubes may be curved.

9. The light field • One might also consider the rays of light passing thru a point. • There are zero such rays. • However, we can define something called radiance. • <see handbook>

10. The light field • Radiant flux: the total energy emitted by a source, or thru some area. • Irradiance: the radiant flux per unit area (radiant flux density). W/m2 • Radiance: the flux density per solid unit angle. (W/m2)/sr, where sr = A/r2. This is effectively the energy passing thru the solid angle of one steradian.

11. The light field • In these terms, the light field is simply the radiance distribution. • I.e., for any point, we can consider the rays of light that impinges upon it from all directions.

12. The light field • Rays • Origins: primary radiators • End: black surfaces • Air, smoke, fog, etc. may also absorb ray energy but such effects are ignored in this treatment.

13. Objects in the light field • Photons are scattered at object surfaces. • Probability of any photon reflecting is a function of the incident angle and the viewing (or exit) direction. • The scattered radiance from a point on a surface, to a particular viewing direction, divided by the irradiance from a given direction, of the surface is called the bidirectional reflectance distribution function (BRDF). • BRDF depends on 4 angular parameters.

14. Objects in the light field • BRDFs tend to be complex functions of the 4 angles. • Many psychophysical studies and computer models ignore this fact. • A perfectly white surface has a BRDF of 1/ (integral)

15. Objects in the light field • Illumination of a matt sphere by directional light • Shading (attitude effect) • Cast shadow • Body shadow • Disc appearance from the direction of illumination: disc with dark edge

16. Objects in the light field • Natural illumination environments • Combination of directional light (solar) and diffuse. • Approximately uniform hemispherical diffuse beam. • Very little body shadow • Ground side will be darker (vignetting) • Ganzfield • Whiteout

17. Photometric effects • Levels of scale • Whole scene • Object • Texture • Incomplete description • Some averaging • Subtexture • Material properties • Incorporated into BRDF

18. Photometric effects • Context • Whole scene • Object • Texture • Subtexture • Examples • Shape from shading affected by contours • Shape from shading with illumination direction information • Scene cues • Specularities • Shadow directions • Degree of diffusion in shadows

19. Photometric effects • Texture and illumination cues • Directional illumination • Texture most apparent near body shadow • Diffuse illumination • Texture due to cracks and pits • Darkness here not due to attitudinal mechanisms but vignetting mechanisms.

20. Photometric effects • Specularities • Light may be scattered by underlying substrates for some angles and reflected for others. • Disappear in diffuse light fields • Directional illumination combined with surface texture can produce arrays of specularities • Example: shining ripples on a lake • Orange vs. tomato

21. Photometric effects • Inter-reflections • Light bounces from one object onto others. • Ties scene together and indicates relations between objects.

22. Photometric effects • Deviations from Lambertian • Backscatter • Asperity scattering • Translucence

23. Photometric effects • Backscatter • Light projected onto textures tends to be reflected back more to the source. • Why? Shadows are not visible to the source. • Viewed from other angles, shadows decrease average luminance. • Attitudinal shading is decreased, making object look flatter.

24. Photometric effects • Asperity scattering • Reflections from hair tips • Common on plants and animals • Furs: hairs are ~parallel with surface provide distinct patterns of specularities. • Asperity occurs when hairs stick straight up. • More light is reflected to the viewer when there are more tips per unit view angle (as near contours). • Again: Attitudinal shading is decreased, making object look flatter. • Can create light edges. • Together with Lambertian shading, explains • The existence of odd order filters in V1 • The use of line drawings in art.

25. Photometric effects • Translucence • Most materials are translucent at the micro scale. • Thus, BRDF concepts do not exactly apply. • Light is not reflected from a point but enters at one point and exits at a slightly different point. • However, if we consider BRDF as reflection from small discs, then micro translucence and microstructures within a surface can explain a BRDF.

26. The structure of pictures • The radiance distribution from any point determines the set of possible picture that can be taken from that point. • However, cameras have their own properties. • Dynamic range • Resolution • MTF • Iimited field of view • Computer screens, film and printers also have their own limitations on dynamic range, resolution etc. • Not mentioned: luminance response functions

27. Shape from shading • Definition: invariant under a group of transformations. • Equivalently: what’s in the scene, rather than what’s in the image. • In this analysis, viewpoint and other parameters are held constant. • Illumination varies.

28. Shape from shading • Concave / convex ambiguities. • Illumination direction / shape confound • Other cues can help resolve these problems. • Only bumps share inter-reflections with the parent plane. • Only dents have internal inter-reflections. • Many such details may be missing in computer graphic implementations, affecting the associated psychophysics.

29. Shape from shading • Cues in collimated light fields • Body shadow • Textural quality of the body shadow edge • Structure of specularities • Structure of contour edge (serrated or not) • Orange vs tomato ( same macro shape )

30. Shape from shading • Cues in diffuse light fields • Overall contour shape • Textural effects due to vignetting • Shading • Due to vignetting not attitudinal • Lambertian is not realistic • Vignetting is not taken into account. • Inter-reflections are not taken into account.

31. Shape from shading • Shape from Lambertian shading • Illumination is assumed directional and the same at all points in the scene • Light returned determines attitude • Surface normals can be considered as unit vectors on the unit sphere (Gauss maps) • Gauss map examples • Plane: point on sphere, degenerate • Cylinders and cones: curves on sphere, degenerate • All others: 2D patches on sphere, 1 to 1 locally • Isophotes • Lines of equal luminance • Correspond to circles on the Gauss sphere

32. Shape from shading • The nature of non 1 to 1 maps • Sphere is multiply covered • Think of a cloth with folds • Folds give rise to critical points (min, max, saddle) in the irradiance field • Folds of the Gauss map correspond to inflections of the surface, parabolic curves • Such parabolic curves bound the convex, concave and saddle regions of the surface • The pattern of such parabolic curves can act as a description of the shape.

33. Psychophysical results • Little definitive • Extreme stimulus reduction • Studying images with only one type of cue (say shading) • Doesn’t provoke strong impressions of shape • Examples • Silhouettes • Line drawings • Indication of shadows and lit areas • The more cues available, the more subjects agree • Is the real problem interaction? • Lack of realism • Published work with computer graphics using unknown or un-described algorithms

34. Open problems • “Majority of literature is irrelevant due to incomplete description of the stimuli, extreme stimuli reduction, or invalid paradigms.” • Stimuli must be complex / natural • Control experiments by using complex / natural images and by varying only a single cue. • Some stimuli are not generic • Ellipsoids have isophotes and shadow boundaries are planar curves. • No other shapes do. • Degenerate Gauss maps • Veridicality • Observer’s response is due to two factors • Idiosyncratic differences • Cues • These are often lumped together.

35. Open problems • How to draw theories from computer vision models for use in human research? • This is difficult because a computer might return a surface mesh. • What is the human equivalent? • What about shading in art? • It differs from actual shading yet we understand it. • Shape impressions of observers need to be recorded rather than simple yes / no forced choices.