Color By Correlation Finlayson et al

1. Color By Correlation Finlayson et al • Goal: identify most likely illumination source for an image. • Was this image lit by daylight, a flourescent light, an incadescent light? • Once I know, I can correct it to a Canonical (standard) illumination. • Note: does not identify where or how bright…just the spectral distribution of the light source.

2. Inputs • Spectral distributions of candidate light sources • Table of probabilities that specific colors (or chromaticities) will occur in any image, by each light source (M matrix) • An Image of unknown illumination • (spectral responsivity of camera)

3. Note: author works in chromaticity rather than color • Removes brightness from color • Chromaticity pair given by: • C = c1, c2, where • c1 = p1/p2 = r/g • c2 = p3/p2 = b/g

4. Author first presents approach intuitively • Then presents a statistically based derivation through the use of baye’s rule: • P(A|b) = P(b|a)*P(a) / P(b)

5. Approach

7. Author then frames the algorithm as a series of Matrix Algebra operations: • thresh(x) produces a vector from x where ith element ==1 ifxi> 0, else ==0 • chist(Cim) produces a histogram of the chromaticities in image im • thresh2(x) produces a vector from x where ith element == 1 ifxi= max of x, else 0 • Let Cill be a 2D vector, where Cilli = chromaticity of illiminant I • Then the chromaticity of the most likely illuminant is given by: • cE = thresh2(thresh(chist(cim))tM ) cill

8. This works because: • thresh(chist(cim)) • produces v, a list of 1’s and 0’s flagging which chromaticities occur at least once in image cim. • thresh(chist(cim))tM • selects those entries in M and adds them up. • thresh2(thresh(chist(cim))tM ) • produces a vector to select the max value from cill

9. After presenting this, he derives a formula for the matrix M through conditional probabilities and Bayes’ rule, and shows that the entries of M can be of the form • Mi,j = ln(P(cioccurs under illuminant j) • This relies on the following: • P( list of v ) occurring in cim is the product of the individual chromaticities occurring (because they are independent) • To maximize the product of the probabilities, we take the log of each probability and maximize the sum of them. • Note that the dot product of two vectors produces the sum of the products of the individual elements.

10. After presenting the algorithm, • a brief description of several other algorithms which attempt to accomplish the same thing. • Frames them in essentially the same way: • a series of matrix multiplications around an M matrix • shows that each method can be seen as a special (simplified) case of his algorithm. • For Example: • the Gray World algorithm is a form of his approach with M replaced by the identity matrix. • 2-D gamut mapping is essentially his algorithm with M thresholded to 1’s and 0’s.

11. Alternate Algorithms • Gray World & Modified Gray World • (modified reduces error from bias of monochrome background) • 3D and 2D Gamut Mapping • (3D works in RGB, 2D in chromaticity) • Sapiro’smethod • based on the Probabilistic Hough Transform • probabilistic approaches • (discussed but doesn’t implement) • Neural Network approach • (3-layer NN, trained on known images & sources)

12. Results

15. Results • Author explains (in Results section) how to get M matrix, as log of probabilities, but: • Example given with positive numbers, 0 where chroma. doesn’t occur. • But, P < 1, so ln(P) < 0 • Ln(0) = -infinity • Example rewards likely, no penalty if not likely • Log penalizes unlikely, rules out light source if impossible value occurs.

16. An Idea for Robot Lighting Invariance • Note: Author uses ability to adjust color to “original” or “neutral” lighting. This suggests an alternative for compensating for lighting in Robot competition: • Calculate segmented color table, correct it to Canonical Light Source. • Look at white (lambertian) object on playing field with each Aibo, to get product of responsivity and light spectrum • Adjust segmented color table inside each Aibo based on color conditions.

17. Questions • Question: It seems that there are many thresholds to set in each of these algorithms. So, setting these thresholds wouldn't make this illimunation problem more complex? • Parameters: • size of chromaticity space • Creating the M matrix • Specifying spectral distributions for light sources • In figure 1 they discuss that they “characterize which image colors are possible under each of our reference illuminates” what does that mean? And how are they characterizing it. • Histogram of chromaticities in a set of images taken under that light source • In the section on Grey-world they say the algorithm “is based on the assumption that spatial average of surface reflectance’s in a scene is achromatic” what does that mean and how did they determine that? • Average color of any image is approx. gray

18. It is indicated in method 3.4 (illuminant color by voting), probabilistic Hough transform is used or it is based of it. What is probabilistic Hough transform? • Would you briefly explain, what are inputs and outputs for neural network approach? • For these, author didn’t provide many details (& didn’t implement them for the paper). Would need to read the references. • It is said that to simplify the color constancy problem a 2D world is considered. Is learning a color in 2D and in 3D the same? • I believe this refers to working in chromaticity rather than tri-color, as this removes brightness dependence. Essentially, allows ignoring the cos(theta) brightness shift so we consider just the ‘color’

19. I do not understand what the importance of the implementation of other color constancy algorithms inside their framework, other than to show the robustness of their design, it just seems redundant. • (?) • Why do they pick the biggest result in Figure 5 (b) as the most possible illuminant? • The figures are just sample results. In fig 5 author wanted to show how well 2D Gamut mapping, the 2nd best algorithm, works.