Computational Vision CSCI 363, Fall 2018 Lecture 17 Computing Motion

1. Computational VisionCSCI 363, Fall 2018Lecture 17Computing Motion

2. Recall: Motion Energy Motion energy filters are constructed with 2 gabor filters, one of which uses a sine and the other uses a cosine (a "quadrature pair"). If you square the outputs of the gabors and sum, the result is motion energy.

3. Motion-Opponency Psychophysical results suggest that neurons in the brain use a motion-opponent processing (e.g. left - right). Evidence: • We cannot see both left and right motion at the same time. If you superimpose a leftward moving sinewave pattern on a rightward moving sinewave pattern, you don't see motion, you see flicker. 2) Motion after-effect: If you adapt to rightward motion, and then look away at static image, you see leftward motion (The waterfall illusion). Demo: http://www.michaelbach.de/ot/mot_adapt/index.html

4. Construction of Motion Opponent energy filters To construct a motion-opponent energy filter, simply subtract the response of a filter tuned to left motion from that of a filter tuned to right motion (or vice versa).

5. Velocity Extraction Problem: Velocity information confounded with contrast. E.g. A weak signal could mean low contrast or low velocity. Solution: Compare the output of different motion channels. (E.g. Left, right and static channels. Change in contrast => Ratio between channels stays the same Change in velocity => Ratio between channels changes.

6. Reverse Phi If a pattern of white and black lines is moved rightward in small steps, people see rightward motion. If the contrast is reversed with each step (white becomes black and vice versa), people see leftward motion. (The Reverse Phi Effect) Demo:https://i.gifer.com/GvPs.gif Energy White energy => rightward motion. Move pattern in steps. Reverse Phi: Move pattern in steps while reversing contrast Dark energy => leftward motion.

7. Fluted Square Wave A square wave that is shifted to the right in 90 deg steps, appears to move right. 90 deg step to right A square wave with the fundamental frequency component removed is a fluted square wave. The highest amplitude component is 3f. When the fluted square wave is shifted to the right in 90 deg steps, it appears to move left!

8. Fluted Square Wave Why? When a square wave that is shifted to the right in 90 deg steps, its fundamental frequency moves right in 90 deg steps. For a fluted square wave, the highest amplitude component is 3f. When the square wave (frequency f) moves 90 deg to the right, the 3f component is being shifted 270 deg to the right, which appears as 90 deg to the left.

9. Energy Response to a fluted square wave x Energy White = Right t Square wave Fluted Square wave Black = Left

10. Moving Plaid Demo Demo of a moving plaid grating: = + Demo: https://www.youtube.com/watch?v=g_sn0WtHK1g

11. tf sfx sfy Motion Energy for 2D images For a 2D image, we use a 3-D gabor filter: Selects frequency range within an ovoid in spatio-temporal frequency space:

12. tf sf tf sfx sfy Velocity lies on a Plane For a 1D image, all measurements of the same velocity lie along a line in SF-TF space (because v = TF/SF) For a 2D image, all measurements of the same velocity lie on a plane in SF-TF space. Find the plane by making multiple measurements and finding best fit.

13. Extra-striate visual areas Folded Cortex Flattened Cortex

14. Dorsal and Ventral Streams

15. Evidence for two processing streams • Evidence for separate streams of processing comes from three areas: • Lesion studies. Lesions in the ventral areas cause selective deficits in color and orientation discrimination abilities. They can also cause deficits in object or face recognition. Lesions in the dorsal areas cause selective deficits in judgments of motion (e.g. speed or direction). Can also cause deficits in localization of objects. 2) Psychophysics: Hard to see motion at "isoluminance". 3)Connection patterns: Parvocellular->4Cb->Superficial cortical layers (color and form) Magnocellular->4Ca->4B-> MT

16. Motion Processing in V1 In V1, some simple cells and complex cells are tuned to direction of motion. I.e. they respond most strongly to motion in a given direction and their response falls off as the motion deviates from that direction. Tuning for 180 deg Firing Rate 120o 240o 180o Direction of Motion Direction Tuning Polar Plot (tuning for zero deg)

17. V1 neurons tuned to temporal frequency V1 neurons appear to be tuned to temporal frequency. Their preferred speed depends on the spatial frequency of the pattern. v = wt/wx Firing Rate Temporal Frequency Neurons in V1 behave like motion energy filters.

18. Motion Processing in MT • MT (The Middle Temporal Area) is thought to be important for processing motion information. • Characteristics of MT neurons: • Cells tuned for direction of motion (more broadly tuned than V1 cells. • Cells tuned for speed. (Some cells specifically tuned for speed. Not dependent on spatial frequency). • Large receptive field sizes. (Some are 100x bigger than V1 receptive fields). They range from 1-2deg in diameter in the foveal region and increase in the periphery.

19. Speed Selectivity McKee and Nakayama have shown that people are very good at discriminating two different speeds independent of spatial frequency. The Weber fraction gives a measure of how big a change in speed is necessary to distinguish two different speeds. It is fairly constant over a broad range of speeds: DV/V = .05 MT may be the area that first computes speed independent of spatial frequency.

20. Direction Selectivity V1 cells are sensitive to the direction of motion of the spatial frequency components of a stimulus. For example, in plaid stimuli a V1 cell will respond when either sine wave is moving in its preferred direction, but will not respond for the pattern motion in its preferred direction. Some MT cells respond to pattern motion. They respond best when the pattern motion of a plaid is in their preferred direction. 20% of MT cells respond to the pattern motion. 40% respond to component motion. 40% are in between.

21. Responses to Plaids Moving plaid: = + Response to pattern motion MT response (20% of cells) V1 response

22. + - Motion opponency MT cells appear to exhibit motion-opponency in their receptive fields: This has implications for motion transparency. - Many MT cells have an inhibitory surround. Motion in the surround inhibits the response to motion in the center. + • The inhibitory surround may be involved in: • Figure-ground segmentation based on motion. • Motion parallax • Heading judgments.

23. Evidence that MT processes motion • Cells in MT prefer moving stimuli to static stimuli. • Lesions of MT cause loss of ability to judge motion direction: Newsome et al. performed an experiment to test this in monkeys. Stimulus: Moving dots--Some percentage move in a coherent direction (correlated dots), the rest move in random directions Task: Judge direction of motion (e.g. up vs. down). Measure: Percent correlation to judge the directions of motion. Result: After lesion of MT, monkeys require a greater percentage of correlated dots to make the discrimination (i.e. they were worse at the motion task). Another piece of evidence for MT being involved in motion comes from experiments in which MT cells are electrically stimulated with a micro-electrode (microstimulation). Salzman & Newsome (1994) showed that they could influence a monkey's perception of motion by stimulation of cells in MT.

24. 2D Motion is just the Beginning 2D image motion contains information about: • Relative depth of surfaces • 3D motion of objects • 3D structure of objects • Direction of observer motion Among other things.