Image Understanding

1. Image Understanding Roxanne Canosa, Ph.D.

2. Introduction • Computer vision • Give machines the ability to see • The goal is to duplicate the effect of human visual processing • We live in a 3-D world, but camera sensors can only capture 2-D information. • Flip side of computer graphics?

3. Introduction • Computer vision is composed of: • Image processing • Image analysis • Image understanding

4. Introduction • Image processing • The goal is to present the image to the system in a useful form • image capture and early processing • remove noise • detect luminance differences • detect edges • enhance image

5. Introduction • Image analysis • The goal is to extract useful information from the processed image • identify boundaries • find connected components • label regions • segment parts of objects • group parts together into whole objects

6. Introduction • Image understanding • The goal is to make sense of the information. Draw qualitative, or semantic, conclusions from the quantitative information. • make a decision about the quantitative information • classify the parts • recognize objects • understand the objects’ usage and the meaning of the scene

7. Introduction • Computer vision uses techniques and methods from: • electronics - sensor technology • mathematics - statistics and differential calculus • spatial pattern recognition • artificial intelligence • psychophysics

8. Low-level Representations • Low-level: little knowledge about the content of the image • The data that is manipulated usually resembles the input image. For example, if the image is captured using a CCD camera (2-D), the representation can be described by an image function whose value is brightness depending on 2 parameters: the x-y coordinates of the location of the brightness value.

9. Low-Level Mechanisms • Low-level vision only takes us to the sophistication of a very expensive digital camera

10. High-level Representations • High-level: extract meaningful information from the low-level representation. • Image may be mapped to a formalized model of the world (model may change dynamically as new information becomes available) • Data to be processed is dramatically reduced: instead of dealing with pixel values, deal with features such as shape, size, relationships, etc • Usually expressed in symbolic form

11. High-Level Mechanisms • High-level vision and perception requires brain functions that we do not fully understand yet

12. Bottom-up vs. Top-down • Bottom-up: processing is content-driven • Top-down: processing is context-driven • Goal: combine knowledge about content as well as context. • Goals, plans, history, expectations • Imitate human cognition and the ability to make decisions based on extracted information

13. Bottom-up v. Top-down Bottom-up? Top-Down? Information flow Information flow

14. Top-down Control Visual Completion:

15. Top-down Control Visual Completion:

16. Top-down Control Visual Completion:

17. Top-down Control Visual Completion:

18. Top-down Control Visual Completion:

19. Old Women or Young Girl? http://dragon.uml.edu/psych/woman.html

20. Expectation and Learning From Palmer (1999)

21. Zolner Illusion Are the black and yellow lines parallel? http://www.torinfo.com/illusion/illus-17.html

22. Visual Illusions Demos http://www.michaelbach.de/ot/index.html

23. The Human Visual System • Optical information from the eyes is transmitted to the primary visual cortex in the occipital lobe at the back of the head.

24. The Human Visual System - 20 mm focal length lens - iris controls amount of light entering eye by changing the size of the pupil

25. The Human Visual System • Light enters the eye through the cornea, aqueous humor, lens, and vitreous humor before striking the light-sensitive receptors of the retina. • After striking the retina, light is converted into electrochemical signals that are carried to the brain via the optic nerve.

26. The Human Visual System image from www.photo.net/photo/edscott/vis00010.htm

27. Multi-Resolution Vision +

28. Multi-Resolution Vision + If you can read this you must be cheating

29. Multi-Resolution Vision • The distribution of rods and cones across the retina is highly uneven • The fovea contains the highest concentration of cones for high visual acuity From Palmer (1999)

30. high Contrast sensitivity low 1 10 100 Spatial frequency (cpd) Contrast Sensitivity

31. Lateral Inhibition

32. Lateral Inhibition

33. 6 6 7 2 3 3 Output perception Lateral Inhibition • A biological neural network in which neurons inhibit spatially neighboring neurons. Architecture of first few layers of retina. 10 10 10 5 5 5 5 10 Input light level -0.2 -0.2 -0.2 -0.2 -0.2 -0.2 -0.2 Layer n +1 +1 +1 +1 +1 +1 Layer n + 1 10-2-2 = 10-2-2 = 10-2-1 = 5-2-1 = 5-1-1 = 5-1-1 =

34. Simultaneous Contrast • Two regions that have identical spectra result in different color (lightness) perceptions due to the spectra of the surrounding regions • Background color can visibly affect the perceived color of the target

35. Simultaneous Contrast

36. Original Painting Task-Oriented Vision Judge their ages Free Viewing Estimate the economic level of the people Remember the clothes worn by the people Guess what they had been doing before the visitor’s arrival

37. Change Blindness • Lack of attention to an object causes failure to perceive it • People find it difficult to detect major changes in a scene if those changes occur in objects that are not the focus of attention • Our impression that our visual capabilities give us a rich, complete, and detailed representation of the world around us is a grand illusion!

38. Change Blindness Demos http://www.usd.edu/psyc301/ChangeBlindness.htm http://viscog.beckman.uiuc.edu/djs_lab/demos.html

39. Modeling Attention • How do we decide where to look next while performing a task? • What factors influence our decision to look at something? • Can we model visual behavior?

40. Input Image Saliency Map Modeling Attention - Saliency Maps • Koch & Ullman (1985), Itti & Koch (2000), Parkhurst, Law, & Neibur (2002), Turano, Geruschat, & Baker (2003)

41. Input Image Computational Model of Saliency color intensity orient Saliency Map center surrounds

42. rods Input Image (RGB) XYZ transform L M S Pre-processing Module Color Map A C1 C2 Intensity Map Oriented Edge Module Object Module G45 G90 G135 G0 Orientation Map Proto-object Map Conspicuity Map

43. rods Input Image (RGB) XYZ transform L M S Color Map A C1 C2 Intensity Map Oriented Edge Module Object Module G45 G90 G135 G0 Orientation Map Proto-object Map Conspicuity Map

44. rods Input Image (RGB) XYZ transform L M S Pre-processing Module Color Map A C1 C2 Intensity Map Object Module G45 G90 G135 G0 Orientation Map Proto-object Map Conspicuity Map

45. rods Color Map Input Image (RGB) XYZ transform L M S Pre-processing Module A C1 C2 Intensity Map Oriented Edge Module G45 G90 G135 G0 Orientation Map Proto-object Map Conspicuity Map

46. Weight Output with Contrast Sensitivity Function CSF = 2.6 (0.0192 + 0.114f) e -(o.114f) ^ 1.1 - Manno and Sakrison (1974) high weight Contrast sensitivity low 1 10 100 Spatial frequency (cpd)

47. Input Image CIE Map Conspicuity Map (C_Map)

48. Verification of Model

49. “Get supplies from the closet” “Work at the computer” “Make a photocopy” Task Differences Free-view

50. Head-Mounted Eye-Tracker Optics module, includes IR source and eye camera Head-tracking receiver LASER Scene camera External mirror - IR reflecting, visible passing