Color-Based Object Identification

1. Color strategies for object identificationQasim ZaidiMarques BosticState University of New YorkCollege of Optometry

2. Color-Based Object Identification Can you identify which parts are sunny or shady? How? Why do you infer illuminant rather than material changes? Can you identify similar foliages from sunny to shady parts?

3. Strategies for color-based object identification Color Constancy: Constancy of subjective appearance by adapting to the illuminant ( Ives, Helson, Smithson & Zaidi etc.). Constancy of Color Contrast/Differences Inverse Optics Methods: Estimate spectrum of illuminant and infer reflectance spectra which are physical invariants (Maloney & Wandell, D’Zmura & Iverson, Brainard & Freeman etc.). Neural algorithms: Based on invariants in cone-catches.

4. Physical invariants for color identification For natural and man-made materials, changes in cone-absorptions of reflected lights for large sets are simply multiplicative, hence rank-orders and cone-contrasts are preserved(Dannemiller; Nascimento & Foster; Zaidi, Spehar & DeBonet).

5. Neural invariants for color identification In post-receptoral cordinates, the color conversion is translational for L/(L+M), and multiplicative for S/(L+M)(Zaidi, Spehar & DeBonet).

6. HEURISTIC ALGORITHM FOR IDENTIFICATION L/(L+M) Identify a subset of objects under equal-energy light with the superset under skylight: Assume that the affine transformation holds, and derive its parameters by matching patterns of chromaticities (Zaidi, 1998). This process should almost always lead to correct identification, which is easy to test.

7. COLOR-DIFFERENCE/CONTRAST STRATEGY Using parallel color difference vectors will lead to correct identifications for many but not all reflectances (Khang & Zaidi). Test this alternative by comparing distractors on lines parallel and orthogonal to the line joining the illuminants.

8. Exp 1: COLOR BASED OBJECT IDENTIFICATION Three of the objects are identical, but the fourth is made from a different paper. The two backgrounds are of the same material, but the two illuminants have different spectra. Which object is made from a different paper than the other three?

9. Number 1 is made from a different paper than the other three objects. Given that neither of the discriminable objects on one side looked exactly like the objects on the other side, what strategy did you follow?

10. Stimuli

11. Legend Each data point is the result of 10 observations O = side correct < 80%, i.e. observer cannot discriminate between standard and test under the same illuminant. O = object correct ≥ 80%, i.e. observer identifies correct objects across illuminants. O = object correct ≤ 20%, i.e. observer misidentifies the standard as the odd object. O = 20% < object correct < 80%, i.e. no reliable identification across illuminants. O = test object under bluish light O = test object under yellowish light Observer: RH – uninformed, experienced Two other observers gave very similar results

12. RESULTS: COLOR BASED OBJECT IDENTIFICATION

13. RESULTS: COLOR BASED OBJECT IDENTIFICATION

14. RESULTS: COLOR BASED OBJECT IDENTIFICATION

15. RESULTS: COLOR BASED OBJECT IDENTIFICATION

16. Object Identification Results Summary • Object identification across illuminants is not perfect, even when tests can be discriminated from the standards. • Because the standard object is a possible choice on every trial, the systematic misidentifications rule out color-constancy, inverse-optics and the affine-heuristic algorithm. • Near the color of the standard, misidentified and correctly identified objects tend to fall on opposite sides, regardless of the color-directions of the distractors. This rules out the color-difference/contrast strategy. • Can a simple perceptual strategy account for the performance results, similar to the strategy for lightness based object identification (Robillotto & Zaidi)?

17. Exp 2: RELATIVE COLOR CATEGORIZATION Standard and test under the same light Instructions: Compared to the standard object, is the test predominantly yellower, bluer, redder, or greener? Observer: QZ – informed, experienced. Three other observers gave very similar results

18. Legend The COLO R of the circle corresponds to observer QZ’s judgment of the test object being predominately yellower, bluer, redder or greener than the standard O = no judgment could be made because the standard and test could not be discriminated O= test object under bluish light O = test object under yellowish light

19. RESULTS: RELATIVE COLOR CATEGORIZATION

20. RESULTS: RELATIVE COLOR CATEGORIZATION

21. RESULTS: RELATIVE COLOR CATEGORIZATION

22. RESULTS: RELATIVE COLOR CATEGORIZATION

23. Summary of Color Categorization Results Standard and test under the same lights* Tests separated into two relative color groups with respect to the standard. The division was similar under the two lights, and corresponded to clusters of correct and incorrect object identification.

24. Similarity based algorithm for object identification across illuminants 1.Identify the illuminant on the test (It) by finding the side on which the two objects have different colors. 2. Compare the colors of the backgrounds to judge the change in color from Is to It. 3. Pick the object most dissimilar to the others along the perceived It - Is color dimension as the odd object.

25. Predictions for tests under the blue/green light Tests perceived as bluer/greener than the standard, will be picked correctly as the odd object.Tests perceived as yellower/redder than the standard, will not be picked as the odd object, unless they are yellower/redder than the standards in the yellow light.Opposite for tests under the yellow/red light

26. Similarity-based Predictions vs Results From QZ’s relative colors to RH’s object identification. I = correct identification, M = misidentification, G = neither Observed Predicted Correct predictions: 92% of identifications & 79% of misidentifications.

27. “Opportunistic” Bayesian Strategy Selection? • Why do observers use color similarity for object identification despite the demonstrable absence of color constancy? • Why are more accurate alternatives not used despite sufficient information in natural scenes? • Is the visual system an “Opportunistic” Bayesian which includes the costs of additional computations in strategy selection?