What is Bayesian inference? How does it apply to perception?

1. Describe the coding of taste and smell signals. Compare both with audition, vision, proprioception, and touch. Describe how attention affects perception. How does attention affect neural responses? What kinds of information are conveyed by the somatosensory system? How is that information encoded in the nervous system? Describe the various ways that optic flow can be used in vision? What is known about sensory prediction? Why is it important?

2. What is Bayesian inference? • How does it apply to perception?

3. x + y = 9 What are x and y? This is an example of an ill-posed problem • problem that has no unique solution

4. Equivalently: X × Y = some cone responses R Example: × = Light Hitting Eye Spectrum of Illuminant Reflectance function of surface Given R, was Y? Question we want to answer: what are the surface properties (i.e., color) of the surface? Perception is also an ill-posed problem (you’d have to know X to make it well-posed)

6. Comparison patch Same light hits the eye from both patches

7. Perception is an ill-posed problem Example #2: ⇒ 3D world 2D retinal image Question: what’s out there in the 3D world? • Ill-posed because there are infinitely many 3D worlds that give rise to the same 2D retinal image • need some additional info to make it a well-posed problem

8. Figure 1. (a) The Necker Cube induces a bi-stable percept. (b) Disambiguation of the bi-stable Necker Cube percept by introducing an occlusion cue and a shadow. (c) An infinite number of 3D configurations could produce the same projection image. Here this fact is illustrated by the cast shadow on the tabletop, but the same projected images would be formed on the eye’s retina.

9. Luckily, having some probabilistic information can help: x + y = 9 Tables showing past values of y: x y Given this information about past values, what would you guess to be the values of x? How confident are you in your answer?

10. from P(A | B) P(B) P(what’s in the world | sensory data) Formula for computing: P(B | A) = P(sensory data | what’s in the world) = “likelihood” P(A) (This is what our brain wants to know!) (given by laws of physics; ambiguous because many world states could give rise to same sense data) & P(what’s in the world) = “prior” (given by past experience) Bayes’ rule • very simple formula for manipulating probabilities conditional probability “probability of B given that A occurred”

11. Examples: Using Bayes’ rule to understand how the brain resolves ambiguous stimuli

12. Let’s use Bayes’ rule: P(A | image) = P(image | A) P(A) P(B| image) = P(image | B) P(B) Which of these is greater? Many different 3D worlds can give rise to the same 2D retinal image The Ames Room A B How does our brain go about deciding which interpretation? P(image | A) and P(image | B) are equal! (both A and B could have generated this image)

13. Which dimples are popping out and which popping in?

14. prior P(OUT | image) = P(image | OUT & light above) × P(OUT) × P(light above) P(IN | image) = P(image | IN & light below ) × P(IN) × P(light below) Which of these is greater? P( image | OUT & light is above) = A P( image | OUT & light is below) = 0 P(image | IN & Light is above) = 0 P(image | IN & Light is below) = A • Image equally likely to be OUT or IN given sensory data alone What we want to know: P(OUT | image) vs. P(IN | image) Apply Bayes’ rule:

15. P(OUT | image) = P(image | OUT & light above) × P(OUT) × P(light above) P(OUT | image) = A × 0.5 × P(light above) Let’s say: “Light above” is 10 times more likely than “light below” P(IN | image) = P(image | IN & light below ) × P(IN) × P(light below) P(IN | image) = A × 0.5 × P(light below) P( image | OUT & light is above) = A P( image | OUT & light is below) = 0 P(image | IN & Light is above) = 0 P(image | IN & Light is below) = A

16. P(OUT | image) = P(image | OUT & light above) × P(OUT) × P(light above) P(OUT | image) = A × 0.5 × P(light above) = 5 × A Let’s say: “Light above” is 10 times more likely than “light below” P(IN | image) = P(image | IN & light below ) × P(IN) × P(light below) P(IN | image) = A × 0.5 × P(light below) = 0.5 × A P( image | OUT & light is above) = A P( image | OUT & light is below) = 0 P(image | IN & Light is above) = 0 P(image | IN & Light is below) = A Bayesian account: “Out” is 10 times more likely!

17. posterior Summary • Perception is an ill-posed problem • equivalently: the world is still ambiguous even given all our sensory information • Probabilistic information can be used to solve ill-posed problems (via Bayes’ theorem) • Bayes’ theorem: • The brain takes into account “prior knowledge” to figure out what’s in the world given our sensory information prior likelihood P(world | sense data) ∝ P(sense data | world ) P(world) (Note that the “posterior” can be considered the prior for the next time step in an ongoing learning process)

18. 1. Use priors 2. Keep around a “full posterior distribution” • we recognize that perception is ambiguous (“ill-posed”), and that the only / best way to deal with that is to use prior information (built up over last 2 minutes, last 2 days, last 2 decades, last 2M years, etc.) • use probabilistic information (don’t just store the “most likely answer” — also store an estimate of our uncertainty) • Ernst, M. & Banks, M. “Humans integrate visual and haptic information in a statistically optimal fashion” Nature, 2002 • Körding, K. & Wolpert, D. “Bayesian Integration in Sensorimotor Learning”Nature, 2004 refs: ref: • Weiss, Simoncelli & Adelson, “Motion illusions as optimal percepts.”Nature Neuroscience (2002) (cue combination) Two take-home facts about “what it means to be Bayesian” in psychology / neuroscience:

19. What does it mean to perceive optimally? • What is a Bayesian model of human perception? • How can we formulate / fit Bayesian models?

21. Visual capture Vision and haptics – view object through a cylindrical lens: vision dominates, but a small effect Of haptic info. Vision and audition – sounds are localized to visual source eg speakers mouth In other instances, senses complement each other – eg feeling an object where there is no conflict, info from back is given by haptics, front by vision. Auditory capture Number of beeps determines whether single or multiple flashes are seen Dominance determined by reliability of the estimate.

23. Figure 3. Visual–haptic size-discrimination performance determined with a 2-interval forced-choice task [29]. The relative reliabilities of the individual signals feeding into the combined percept were manipulated by adding noise to the visual display. With these different relative reliabilities four discrimination curves were measured. As the relative visual reliability decreased, the perceived size as indicated by the point of subjective equality (PSE) was increasingly determined by the haptic size estimate (haptic standard, SH) and less by the visual size estimate (visual standard, SV). This demonstrates the weighting behaviour the brain adopts and the smooth change from visual dominance (red circles) to haptic dominance (orange triangles). As shown, the PSEs predicted from the individual visual and haptic discrimination performance (larger symbols with black outline) correspond closely to the empirically determined PSEs in the combined visual–haptic discrimination task. (JND . just noticeable difference.)

24. (better to say “act”) What does it mean to perceive optimally? • Our senses provide noisy measurements of the environment (e.g., “something moving in the grass”) • We have some well-defined (deterministic) cost function describing the cost of making different errors (e.g., cost saying “wildebeast” when the true stimulus was “lion”) • We have some beliefs about the underlying probability of different stimuli (e.g., “wildebeast” more common than “lion”) General Goal: Decide: “what stimulus is present?” Assumptions:

25. (better to say “act”) What does it mean to perceive optimally? • Our senses provide noisy measurements of the environment (e.g., “something moving in the grass”) • We have some well-defined (deterministic) cost function describing the cost of making different errors (e.g., cost saying “wildebeast” when the true stimulus was “lion”) • We have some beliefs about the underlying probability of different stimuli (e.g., “wildebeast” more common than “lion”) Solution: Given a noisy measurement, pick stimulus that minimizes the expected cost. Assumptions:

26. (better to say “act”) What does it mean to perceive optimally? “Bayesian Decision Theory” • Our senses provide noisy measurements of the environment (e.g., “something moving in the grass”) • We have some well-defined (deterministic) cost function describing the cost of making different errors (e.g., cost saying “wildebeest” when the true stimulus was “lion”) • We have some beliefs about the underlying probability of different stimuli (e.g., “wildebeest” more common than “lion”) Solution: Given a noisy measurement, pick stimulus that minimizes the expected cost. Assumptions: noise distribution / likelihood cost function prior

27. What is the true sensory noise? (Or impose noise that swamps internal noise.) What is the true cost of saying “wildebeest” when it’s a lion, vs. saying “lion” when it’s a wildebeest? (Or impose costs: e.g., pay subjects) Measure the relative population size of wildebeests and lions (Or impose prior in an artificial task). Two basic approaches: 1. “Objective” approach: measure these three ingredients in the real world. Determine if observers are in fact optimal. (“Bayesian ideal observer analysis”) noise distribution / likelihood cost function prior • Definitive answer: “yes” or “no”. Can quantify how close observers are to optimal

28. Estimate this! Assume to be Gaussian (estimate variance) Assume some standard choice (e.g., mean-squared error) Two basic approaches: 2. “Subjective” approach: measure stimuli and observer responses. Is there some setting of these three under which observer can be said to be Bayes optimal? Under-constrained problem in general - have to make some assumptions noise distribution / likelihood cost function prior • Not clear if the answer is ever “no”. (Would anyone publish a null result?) • However: do the noise, cost and prior make sense? Is it reasonable to think that this is what subjects are really doing? (Offers parsimonious explanation of percepts)