Shapelets Correlated with Surface Normals Produce Surfaces

1. Shapelets Correlated with Surface Normals Produce Surfaces Peter Kovesi School of Computer Science & Software Engineering The University of Western Australia

2. Shape from Shading and Shape from Texture attempt to • Calculate the surface normal direction at each point in the image, and then • Infer the surface shape from the surface normals.

3. N slants Surface normals are usually calculated in terms of slant and tilt eye surface Slant is the angle between the viewing direction and the surface normal

4. Tilt is the angle of the projection of the surface normal on a plane perpendicular to the viewing direction projection of surface normal tilt t N slant s plane perpendicular to viewing direction viewer

5. Problem: Tilt often has an ambiguity ofp

6. Problem: Tilt often has an ambiguity ofp

7. Problem: Regularization schemes are often used to deal with noise. However, they typically involve the minimization of an Energy Term that has mixed units surface gradient error (highly nonlinear with surface angle) shading error bending energy

8. Problem: Regularization schemes are often used to deal with noise. However, they typically involve the minimization of an Energy Term that has mixed units surface gradient error (highly nonlinear with surface angle) shading error bending energy • The weighting parameters have to act as unit conversion operators. • The appropriate values will be dependent on scale. • This can be problematic if the regularization is being applied within a multi-scale framework. • Discontinuities have to be automatically detected and treated specially.

9. Alternative approach: Project the surface gradients onto a set of integrable basis functions and reconstruct the surface from these. Frankot and Chellappa [1988] first introduced this approach using Fourier basis functions. • Features of work presented here: • Localized, non-orthogonal basis functions are used. • Correlations with basis functions are performed with respect to slant and tilt rather than in terms of gradient in x and y. • Ambiguities of  in tilt can be considered. • Low quality reconstructions are possible with slant information alone.

10. Decomposing and Reconstructing Signals with Basis Functions signal band-pass filters/wavelets filtered at scale 2 filtered at scale 4 filtered at scale 6 Reconstruction obtained by summing band-passed signals over 6 scales

11. We only have the gradient of the surface…

12. We only have the gradient of the surface… Differentiation is linear surface gradient surface band-pass filter

13. We only have the gradient of the surface… Differentiation is linear A band-passed version of the signal can be obtained from a correlation performed between the gradient of the signal and integral of the band-pass filter. surface gradient Integral of band-pass filter (shapelet gradient) surface band-pass filter

14. Choice of Shapelet • The gradient of shapelet function must satisfy the admissibility condition of zero mean. • Must ensure preservation of phase information in the signal. • The bank of shapelet filters should provide a uniform coverage of the signal spectrum so that it is faithfully reconstructed. These requirements are satisfied by a Gaussian

15. Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient.

16. Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient. The signal gradient spectrum differs from the original signal spectrum in that its amplitudes are scaled by the frequency.

17. Note that correlating the gradient of one shapelet filter with the gradient of the signal will correspond to extracting a band of frequencies from the signal gradient. The signal gradient spectrum differs from the original signal spectrum in that its amplitudes are scaled by the frequency. The sum of the shapelet gradient transfer functions should form a curve that is inversely proportional to frequency to counteract the scaling of the gradient spectrum. The net result will then be an even coverage of the original signal spectrum.

18. Shapelet bank formed by five Gaussians with height proportional to scale. Transfer functions of the gradients of the shapelets above. The sum forms a curve that is approximately inversely proportional to frequency.

19. A bank of shapelets will not form an orthogonal basis set This does not matter if we have a strongly redundant basis set. (Achieved by using a continuous wavelet transform.) A non-orthogonal basis set can be considered to be the sum of several orthogonal basis sets. - We get accurate reconstruction up to a scale factor

20. The scaling of the reconstruction due to the redundancy of the basis set can be determined and corrected if required. • Overcomplete basis sets have the advantage that small changes in local features result in correspondingly small changes in the basis coefficients. • Overcompleteness also provides robustness to noise.

21. In Summary: • Correlate the gradients/normals of the surface with the gradients/normals of a set of basis functions. • These correlation results correspond to bandpassed versions of the original signal. • Reconstruct surface by summing correlation results. • Perform correlation in terms of slant and tilt separately • - slant and tilt are calculated by different means. • - slant and tilt have differing degrees of uncertainty.

22. Correlation of Slant Slant lies in the range 0 -p/2which only provides gradient magnitude information. The slant correlation at scale i is formed by With no tilt information this correlation measure matches positive and negative gradients equally. (No distinction between concave or convex shape) gradient of shapelet at scale i gradient of surface

23. Correlation of Tilt • If gradient magnitudes match and tilts match • correlation is +ve • If gradient magnitudes match and tilts differ by p • correlation is -ve • If tilts differ by p/2 correlation is 0 • Use the cosine of the tilt difference as a correlation measure tilt of shapelet at scale i tilt of surface

24. Overall Correlation Reconstruction

25. synthetic test surface needle diagram of surface normals

26. surface normals reconstruction over 2 scales reconstruction over 4 scales reconstruction over 6 scales

27. Shapelet reconstruction using the gradients of the range data Range data from OSU database

28. Slant and tilt with Gaussian noise of 0.3 radians standard deviation Shapelet reconstruction Terzopoulos multigrid reconstruction Frankot Chellappa reconstruction

29. Allowing for Tilt Ambiguity of p When tilts are aligned value is 1 When tilts are orthogonal value is 0 When tilts are opposite value is 1 … results in rectification of the band-pass components of the surface

30. Reconstruction with a tilt ambiguity of p bumps surface reconstruction of ramps surface reconstruction

31. Reconstruction with slant data alone Slant-only reconstruction of bumps Slant-only reconstruction of ramps Sinc function Slant-only reconstruction of sinc

32. …after segmentation and negation Slant-only reconstruction of face

33. Real data: Surface normals from texture Shapelet reconstruction

35. Shapelet reconstruction Sand ridges: Slant set to /4 at all points Tilt set to 0 for white regions,  for black regions

36. Shape from Occluding Contours Surface normals at occluding contours are perpendicular to viewing direction. This is a powerful cue to shape. Manually marked contours

37. A crude approximation of the surface normals in the scene… At all points slant is set to 0, except along contours where slant is set to , and tilt is set to be perpendicular to contour. This gradient field is not integrable.

38. Shapelet reconstruction

40. Manually drawn contours

45. Conclusions • Reconstructions from surface normals using shapelets is simple and the process is robust to noise. • The use of basis functions implicitly imposes a continuity constraint in the reconstruction. • Being able to treat slant and tilt separately permits data with tilt ambiguity, or no tilt data, to be considered. • Much can be made from limited amounts of surface normal data. This allows some some aspects of structure to be deduced from line drawings.