410 likes | 505 Views
Image Stabilization by Bayesian Dynamics. Yoram Burak Sloan-Swartz annual meeting, July 2009. What does neural activity represent?. In Bayesian models: probabilities. Accumulated evidence in area LIP Shadlen and Newsome (2001). Direction of motion: single, static variable.
E N D
Image Stabilization by Bayesian Dynamics • Yoram Burak • Sloan-Swartz annual meeting, July 2009
What does neural activity represent? • In Bayesian models:probabilities Accumulated evidence in area LIP Shadlen and Newsome (2001) Direction of motion: single, static variable
What does neural activity represent? • In Bayesian models:probabilities Accumulated evidence in area LIP Shadlen and Newsome (2001) Direction of motion: single, static variable What about multi-dimensional, dynamic quantities?
Foveal vision and fixational drift Fixational drift is large in the fovea: - between micro-saccades - ~20 receptive fields - between spikes (100 Hz) - ~2-4 receptive fields ! Image from: X. Pitkow cone separation: 0.5 arcmin
Foveal vision and fixational drift Fixational drift is large in the fovea: - between micro-saccades - ~20 receptive fields - between spikes (100 Hz) - ~2-4 receptive fields ! Image from: X. Pitkow cone separation: 0.5 arcmin Downstream areas require knowledge of trajectory to interpret spikes
Joint decoding of image and position Bayesian: vs. N x 2probabilities Discrimination task: X. Pitkow et al, Plos Biology (2007) # positions
Joint decoding of image and position Bayesian: vs. N x 2probabilities Discrimination task: X. Pitkow et al, Plos Biology (2007) # positions 30 x 30 binary pixels Unconstrained image N x 2900probabilities
Joint decoding of image and position Bayesian: vs. N x 2probabilities Discrimination task: X. Pitkow et al, Plos Biology (2007) # positions 30 x 30 binary pixels Unconstrained image N x 2900probabilities Can the brain apply a Bayesian approach to this problem?
Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?
Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?
Decoding strategy Factorized representation: Discards information about correlations
Decoding strategy Factorized representation: Discards information about correlations Update dynamics: minimize DKL evidence, diffusion Exact if trajectory is known.
Decoding strategy Factorized representation: Discards information about correlations Update dynamics: minimize DKL evidence, diffusion Exact if trajectory is known. Retinal encoding model: evidence - Poisson spiking (rate λ1 for on pixels, λ0 for off) diffusion - Random walk (diffusion coefficient D)
Decoding strategy Factorized representation: Discards information about correlations Neural Implementation - Two populations: where , what For 30 x 30 pixels: N × 2900 → N + 900quantities.
Update rules Update of what neurons: Ganglion cells multiplicative gating What nonlinearity Where
Update rules Update of what neurons: Ganglion cells multiplicative gating What nonlinearity Where Update of where neurons: Ganglion cells multiplicative gating Where + diffusion What
Demo m x m binary pixels image retina 2d diffusion (D) Poisson spikes: 100 Hz (on), 10 Hz (off) Decoder
Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?
Performance Performance degrades with larger D (and smaller λ) accuracy Convergence time [s] D D
Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D
Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D
Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D
Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D
Performance scales with linear image size m accuracy Convergence time [s] D/m D/m m x m pixels
Performance scales with linear image size m D* accuracy Convergence time [s] D/m D/m m x m pixels Analytical scaling:
Performance Performance improves with image size. Success for images 10 x 10 or larger Prediction for psychophysics: Degradation in high acuity tasks when visual scene contains little background detail.
Temporal response of Ganglion cells Common view: fixational motion important to activate cells, due to biphasic response f(t) 50 ms t Temporal response makes decoding much more difficult. Non-Markovian: Need history
Temporal response of Ganglion cells Approach: Choose decoder that is Bayes optimal if the trajectory is known. Ganglion accuracy Convergence time [s] Where What D D history dependent decoder / naive decoder “filtered trajectory”
Temporal response of Ganglion cells Is fixational motion beneficial? Known trajectory , perfect inhibitory balance Convergence time [s] D Optimal D - order of magnitude smaller than biological value
Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?
Network architecture Each ganglion cell innervates multiple what & where cells (spread: ~10 arcmin) Reciprocal, multiplicative gating Ganglion What Where
Activity: What neurons Slow dynamics, evidence accumulation Where neurons Fewer. Highly dynamic activity Tonic, sparse in retinal stabilization conditions.
Activity: What neurons Slow dynamics, evidence accumulation Where neurons Fewer. Highly dynamic activity Tonic, sparse in retinal stabilization conditions. Where in the brain? Monocular If so, suggests LGN or V1 LGN? Modulatory inputs to relay cells (gating?) V1? Lateral connectivity in where network, Increase in number of neurons.
Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference
Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference Explicit representation of stabilized image “What” and “where” populations
Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference Explicit representation of stabilized image “What” and “where” populations Good performance at 1 arcmin resolution Problem is easier for large images, for coarser reconstruction
Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference Explicit representation of stabilized image “What” and “where” populations Good performance at 1 arcmin resolution Problem is easier for large images, for coarser reconstruction Network architecture: Many-to-one inputs from retina, multiplicative gating (what/where)
Acknowledgments Uri Rokni Haim Sompolinsky Markus Meister Special thanks - the Swartz foundation