Lecture 24: More on Reflectance

1. Lecture 24: More on Reflectance CAP 5415

2. Recovering Shape • We’ve talked about photometric stereo, where we assumed that a surface was diffuse • Could calculate surface normals and albedo • What if the surface isn’t diffuse?

4. Basic Idea • Assume illumination is distant • Illumination only depends on orientation of the point on the surface • A sphere can tell us the illumination at every surface normal (Image from Ron Dror)

5. Basic Idea • How can I tell the orientation of every point on the bottle?

7. Results from the bottle

8. Key Advantage • Can work with materials that are difficult to model mathematically

9. Estimating Reflectance • This method works well • When a probe is available • If a probe is not available, then we have to use the reflectance of the object • How can we recover the reflectance?

10. One option: Use the statistics of the real world • Every image is a picture of the illumination of the real world • Like images, the histograms of illuminations have a regular appearance (From Dror and Adelson)

12. Basic method looks at the response of steerable-pyramid responses In some sense, associates different materials with different textures%

13. Works surprisingly well

14. 94% Accuracy on these materials Similar to human performance Classified based on a number of histogram measurements (percentile values, moments, etc.)

15. Limitations • Works on objects of known geometry • What if we have an arbitrary scene? • How can we solve the problem then? • Make a simplifying assumption: • Surfaces are Lambertian • Now we only need to recover the albedo

16. View the image as shading and albedo This is shading These stripes are reflectance changes

17. Goal: Find the albedo Take this image Observation Shading Information Albedo Information By splitting it into two images that represent the shading and albedo of the scene

18. Why? • The effects of these characteristics are combined together in the image • To understand this image, we need to be able to separate the effects of these characteristics

19. Other reasons • Ability to reason about shading and reflectance independently is necessary for most image understanding tasks. • Material recognition • Image segmentation • Want to understand how humans might do the task. • An engineering application: for image editing, want access and modify the intrinsic images separately • Intrinsic images are a convenient representation. • More informative than just the image • Less complex than fully reconstructing the scene

20. Your visual system is trying to solve a similar problem (From Edward Adelson) The squares marked A and B have the same intensity

21. What is shading? • Shading is the interaction of the shape of the surface and the illumination • Shading image should look like gray plaster

22. Albedo • Albedo – Fraction of light reflected at a point • Albedo image represents the albedo at every point on the surface • Points inside the squares should reflect less light

23. An image is formed as x Shading Image Reflec Reflec Reflec Observed Image Albedo Image

24. Our goal is to invert the process • Using images to represent characteristics that are intrinsic to the scene’s appearance • Known as intrinsic image representation[Barrow and Tenenbaum 1978] Observed Image Shading Image Albedo Image

25. Estimating Shading and Albedo Images

26. First, Simplified Model of the World • Albedo image –Patches of constant albedo • Called Mondrian Images (Land and McCann) • Shading image – Smooth Albedo Image Shading Image Observation

27. Given this observation, what’s the albedo image? Observation

28. Estimation in 1-D • It’s easy to see how to estimate the albedo in 1-D • We’ll work in the log domain, so the problem is additive Image Intensity Column in the image

29. Now compute the derivative • The locations of the changes in the albedo image are obvious Log Image Intensity Log Image Intensity Column in the image Column in the image

30. Eliminate the derivatives caused by shading changes Set everything but these derivatives to 0, then re-integrate Log Image Intensity Log Image Intensity Column in the image Column in the image Estimate of the log albedo image Known as the Retinex Algorithm [Land and McCann]

31. Basic Strategy for Estimating Images Pseudo-Inverse Classify Every Derivative as Shading or Albedo Change Estimate of the horizontal derivative Observation Estimate of the Albedo Image Similar to Horn’s Implementation Estimate of the vertical derivative

32. Where Retinex Breaks Down • Unfortunately, this assumption is often broken in the real-world

33. Retinex • Most other work on this problem still uses the same basic assumption (Kimmel 03) • Researchers have also tried augmenting the input • Use multiple images, like time-lapse [Weiss 2001, Matsushita 2003] • Color (Funt et al.) • Extra information may not be available • Estimating shading and albedo from a single, gray-scale image is a good test our ability to model images and surfaces

34. Beyond Mondrians • Can go beyond the simple world of Mondrian images in two ways: • Learn better models of local image statistics • Look at an image patch rather than just a single derivative value • Propagate information across the image

35. Why Propagate? • One of these patches is an albedo change and the other is shading

36. Recovering Intrinsic Images • Classify each x and y image derivative as being caused by either shading or a reflectance change • Recover the intrinsic images by finding the least-squares reconstruction from each set of labeled derivatives. (Fast Matlab code for that available from Yair Weiss’s web page.) Classify each derivative (White is reflectance) Original x derivative image

37. Outline of our algorithm • Gather local evidence for shading or reflectance • Color (chromaticity changes) • Form (local image patterns) • Integrate the local evidence across space. • Assume a probabilistic model and use “belief propagation”. • Show results on example images

38. Probabilistic graphical model Unknown Derivative Labels (hidden random variables that we want to estimate)

39. Local Color Evidence Some statistical relationship that we’ll specify Derivative Labels Probabilistic graphical model • Local evidence

40. Derivative Labels Local Color Evidence Probabilistic graphical model • Local evidence Local Form Evidence

41. Probabilistic graphical model Propagate the local evidence in Markov Random Field. This strategy can be used to solve other low-level vision problems. Local Evidence Hidden state to be estimated Influence of Neighbor

42. Local Color Evidence For a Lambertian surface, and simple illumination conditions, shading only affects the intensity of the color of a surface Notice that the chromaticity of each face is the same Any change in chromaticity must be a reflectance change

43. Red Red Blue Blue θ Green Green Classifying Color Changes Intensity Changes Chromaticity Changes Angle between the two vectors, θ, is greater than 0 Angle between two vectors, θ, equals 0

44. Color Classification Algorithm • Normalize the two color vectors c1 and c2 • If (c1c2) < T • Derivative is a reflectance change • Otherwise, label derivative as shading c1 c2

45. Result using only color information

46. Results Using Only Color • Some changes are ambiguous • Intensity changes could be caused by shading or reflectance • So we label it as “ambiguous” • Need more information Input Reflectance Shading

47. Utilizing local intensity patterns • The painted eye and the ripples of the fabric have very different appearances • Can learn classifiers which take advantage of these differences