Specularity, the Zeta-image , and Information-Theoretic Illuminant Estimation

1. Specularity, the Zeta-image, and Information-Theoretic Illuminant Estimation Mark S. Drew1, Hamid Reza Vaezi Joze1, and Graham D. Finlayson2 2School of Computing Sciences, The University of East Anglia Norwich, England graham@cmp.uea.ac.uk 1School of Computing Science Simon Fraser University Vancouver, BC, Canada mark@cs.sfu.ca

2. Zeta-Image: Goal= Discover the light color (yet another!??) Relative Chromaticity: The main idea is that we can get at a good solution for the chromaticity of the light by dividing image chromaticity 3-vector  by the candidate light chromaticity e — the “relative chromaticity”  =  e [where is component-wise division].

3. Algorithm: Then we show that, over pixels that are specular or white, the log of the relative chromaticity log() is perpendicular to the light chromaticity e in color space. This gives a useful hint for recovering e.

4. “Neutral interface”: Lee, JOSA 1986, “Method for computing the scene-illuminant hromaticity” k = R,G,B surface specularity pixel color light Light is “white enough”: Borges, JOSA 1991, “Trichromatic approximation method for surface illumination” Proof; Background: Simple image formation model --

5. Proof…  = {R,G,B}/{R+G+B) Now chromaticity: Relative chrom.:

6. Proof… …Relative chrom.: simple!

7. Proof… Now let’s head for a Planar Constraint:

8. Proof…

9. Therefore, let the Zeta-image† be • Definition: where e is the chromaticity of illuminant and (x) is the chromaticity value of pixel at position x . Properties • It has the structure of the Kullback-Leibler Divergence from Information Theory; • Zeta is low (near-zero) at specularities: †Patent applied for

10. Zeta-image for illumination estimation , == Search • Planar constraint:For near-specular pixels or white surfaces, Log-Relative-Chromaticity values are orthogonal to the light chromaticity. Or, equally, the zeta image is near zero. • So the best light chromaticity e for an image is that which minimizes the zeta-image for near-specular pixels (or white surfaces) => guess a domain  ; then

11. Search — explained: • assume candidatee; form ; the lowest 10-percentile, say, of dot-product values could be near-specular pixels. • Over a grid of possible light-chromaticitiese,minimize dot-product values • over candidate illuminants for those lowest 10-percentile pixels.

12. Or, Thm:Analytic solutionis the geometric mean of where is a set of bright pixels Search is better than Analytic but slower.

13. Domain  for Analytic Solution: • We start by approximating  as top-5% brightness pixels. • Could be any other method to indicate near- specular pixels and white surface regions. • Detecting Failure  i.e., detecting images not having specularity or white surfaces in top brightness pixels: •  can stem from areas of images belonging to the brightest surface which happens to tend to be some particular surface color. • we can simply check if these pixels are in the possible chromaticity gamut of illuminants. •  can be a bag-of-pixels from all over the image. • we can investigate the distribution of  in chromaticity space.

14. Does this work?... divide by correct light, bottom 20% of Zeta: yes!

15. ...Does this work?... incorrect light, bottom 20% of Zeta: no! correct light, float (inverted) Zeta 

16. Details of Analytic: How did we come to the geometric mean? Solve:  

17. …Details of Analytic… How do we know  is positive? If components were probabilities then has structure of Kullback-Leibler Divergence: extra bits to code samples from ek when using codebook based on ik , so positive!

18. …Details of Analytic… Final step: form  using the geomean chromaticity for bright pixels, and then trim to least-10% values of Zeta, and recalculate the geomean. Does this work?

19. float (inverted) Zeta bright lowzeta = bright & (zeta<quantile(zeta)),0.10);

20. plots of  for correct light e(O)and for analytic solution e(x) :

21. In expts. mask off the ColorChecker

22. so far: Algorithm 1: Use either analytic answer, or a simple hierarchical grid search over light- chromaticity Algorithm 2: Planar Constraint Applied as Post-Processing Use any alg’s answer for e; take SVD of lowest-10% dot-product pixels: improves light estimate!

23. fallback position: If either of two chromaticity checks fails, use Grey-Edge.

24. Post-processing: Results.

25. Illumination Estimation: Results. (Analytic)

26. Thanks! Funding: Natural Sciences and Engineering Research Council of Canada