1. Optimizing Monkeys & Optimistic Math

1. Progress in MURI 15 (2007- 09)Mathematical modeling of decision behavior.AFOSR, Alexandria, VA, Nov 17th, 2009Phil Holmes 1. Optimizing monkeys? Balancing noisy stimulus information and reward expectations (Feng, Holmes, Rorie, Newsome, PLoS Comp. Biol. 5 (2), 2009). 2. Time-dependent perturbations can distinguish integrator dynamics (Zhou, Wong-Lin, Holmes, Neural Comput., 21, 2336-2662, 2009). [3. Mean field reduction of a spiking neuron integrator model with neuromodulation (Eckhoff, Wong-Lin, Holmes, SIAM J. Appl. Dyn. Sys., in review, 2009). Not to be presented today. ] -----------------------

2. 1. Optimizing Monkeys&Optimistic Math

3. (actually, nonlinear DD/OU processes) via strong attraction to a 1-d subspace: Leaky competing accumulators reduce to OU (Usher & McClelland, 1995,2001) Subtracting the accumulated evidence yields an OU process (linearized DD case shown here). This generalizes to nonlinear systems: stochastic center manifolds.

4. Fixed viewing time (cued response) tasks can be modeled by DD/OU processes. We consider the PDF of sample paths of the SDE, which is governed by the forward Fokker-Planck or Kolmogorov PDE: Interrogate solution at cue timeT: is ? [Neyman-Pearson] General solutions for time-varying drift (SNR) are available, so we can predict accuracy (psychometric functions):

5. DD can incorporate priors: reward expectations Integrating bottom-up (stimulus) and top-down (prior expectation) inputs: a motion discrimination task with four reward conditions. Introduces reward priors (expectations). HL LH HH Why monkeys? Electrophysiology! (Rorie, Reppas, Newsome) LL We only model this part Reward cue Motion A. Rorie & W.T. Newsome, SfN 2006; in prep., 2008).

6. DD/OU models with reward bias priors, 1 Choose a model for reward expectation. Two examples are: 1. Initial condition set at start of motion period: motion period only. 2. Bias applied to drift rate throughout reward cue and motion periods: reward cue period motion period. for predicts shifted PMFs: HL HH LL LH Leak-dominated Balanced Inhibition-dominated (stable OU, recency) (DD, optimal) (unstable OU, primacy) Feng, H, Rorie, Newsome, PLoS Comp. Biol.5 (2), 2009.

7. DD/OU models with reward bias priors, 2 • These reward expectation models both lead to PMFs that can be written in the simple form • whereb1acuity= ability to • use signal (SNR) • b2shift = estimate of • reward bias • depending on the model chosen. But unless we have a range of “priming” and viewing times* , the models can’t be distinguished on behavioral evidence alone: the parameters cannot be separated. [*But see McClelland presentation.] • We can nonetheless compute shifts that maximize expected rewards, • e.g., in simple case of fixed coherence : • (More complex expressions hold for • the mix of coherences actually used.) reward ratio ~1/SNR

8. DD/OU models with reward bias priors, 3 Fit the model to the Rorie-Newsome behavioral data from two adult male rhesus monkeys, averaged over all sessions: monkey A monkey T For fixed the PMFs all reduce to a 2 parameter family: slope shift

9. DD/OU models with reward bias priors, 4 Maximize expected rewards: compute optimal shifts given animals’ slopes and compare with their actual shifts: reward ratio sums of Gaussians Both animals overshift, and T prefers alternative 2 when rewards are equal. But does this cost them much? How steep is the reward hill?

10. DD/OU models with reward bias priors, 5 Expected reward functions are rather flat, and overshifting costs less than undershifting. So they don’t lose much! 99.5% performance bands monkey A monkey T

11. DD/OU models with reward bias priors, 6 Overall performance is good, although acuity (= slope b1) and shifts (b2) vary significantly from session to session. In spite of “visual correlation,” the animals do not exhibit significant b1 vs b2 correlations (performance tuning). Monkey A: Blue is optimal; magenta curves are 99% & 97% of optimal. With few exceptions, A stays inside 97% in every session. Monkey T is not as good, shows bias to T2.

12. Conclusions 1 • DD processes extend to include top-down cognitive control: can be used to predict optimal PMFs. Two subjects tested both overshift, but garner 98 − 99% of maximum possible rewards, in spite of significant session-to-session variability. App of rigorous math methods. • Brief impulsive perturbations during stimulus presentation can reveal the dynamics of integrators, distinguish pure DD processes from leak- and inhibition-dominated integrators*, and even from nonlinear model. (*stable, unstable OU processes)

13. 2. Brief perturbations can distinguishing integrators

14. Drift-diffusion and OU processes with thresholds can model RT tasks. Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

15. Time-dependent perturbations can distinguish integrator types. Mean RT Variance threshold for choice 1 time Decision variable x threshold for choice 2

16. Perturbation modifies RT distributions Perturbn must be early to have strong effect! means standard deviations Means and standard deviations of RT distributions are shifted in distinct manners for constant and rising drift rates (CD, TD) and for stable and unstable OU processes (SOU, UOU); positive pulse +, negative pulse _. Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

17. Zero-effect perturbations CD, TD, SOU and UOU are most easily distinguished by tuning a neg-pos pulse pair so that the mean RT is not changed. These employ the recency/primacy effects previously noted by Usher-McClelland. The “simple” patterns seen here occur only for low noise and early perturbations (pert time T << mean RT; effects decay as T increases). For high noise the mean and SD patterns are less clear. Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

18. Further results and related work The ideas extend to nonlinear systems (subcritical pitchfork case), although for effects are harder to interpret. Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009. Gao-McClelland have replicated a slight decrease in the effect of perturbations delivered later, under the cued response (interrogation) protocol. Also, subjects appear more responsive to coherent motion early in the trial than later. See Gao presentation, and cf. fits of OU and variable drift DD models to data from J. Gold lab (Penn). (Eckhoff, H, Law, Conolly, Gold, New J. Phys.10, 2008.) Perturbation experiments are planned (Usher, McClelland, Simen, …)

19. Conclusions 2 • DD processes extend to include top-down cognitive control: it can predict optimal PMFs. Two subjects tested both overshift, but garner 98 − 99% of their maximum possible rewards, in spite of significant session-to-session variability. App of rigorous math methods. • Brief impulsive perturbations during stimulus presentation can reveal the dynamics of integrators, distinguish pure DD processes from leak- and inhibition-dominated integrators*, and even from nonlinear model. (*stable, unstable OU processes)

21. The drift-diffusion (DD) process, a cornerstone of 20th century physics, and perhaps the simplest stochastic ODE, is an optimal decision maker for two-alternative forced choice perception tasks with noisy data: drift rate noise strength Here is the accumulated evidence (the log likelihood ratio): when it reaches the threshold , declare R or L the winner. DD is a continuum limit of SPRT (Wald, 1947). The model has only 2 parameters: Behavioral data,neural recordings in primates, fMRI and EEG in humans (MURI 16) andspiking neuron modelssupport the contention that evidence is accumulated in this manner by cortical networks. Preamble: optimal choices

22. Behavioral evidence: RT distributions Human reaction time data in free response mode can be fitted to the first passage threshold crossing times of a DD process. Prior or bias toward one alternative can be implemented by setting starting point . Extended DD: variable drift rates & starting points. Simple expressions for mean RT, accuracy.Ratcliff et al., Psych Rev. 1978, 1999, 2004 Simen et al., J. Exp. Psych.:HPP, in press, 2009. thresh.+Z drift A thresh.-Z

23. Working hypothesis: motion sensitive cells in visual cortex MT pass noisy signals on to LIP, FEF, … where integration occurs. Spikingneurons reduce to leaky accumulators threshold LIP MT Simulation, analysis and reduction of spiking neuron models: X.-J. Wang, Neuron, 2002; K-F. Wong & X.-J. Wang, J. Neurosci.,2006. Extension to model NE modulation of LIP: P. Eckhoff, K-F. Wong, PH. J. Neurosci, 29 (13),2009; P. Eckhoff, K-F. Wong, PH. (in review,2009). Experimental observations: K.H. Britten, M.N. Shadlen, W.T. Newsome, J.D. Schall & A. Movshon (various papers, 1992-2009).

24. Neural evidence: firing rates Spike rates of neurons in cortical areas rise during stimulus presentation, monkeys signal their choice after a threshold is crossed (like DD process). thresholds J. Schall, V. Stuphorn, J. Brown, Neuron, 2002. Frontal eye field (FEF) recordings. J.I Gold, M.N. Shadlen, Neuron, 2002. Lateral interparietal area (LIP) recordings.

25. Accuracy for fixed viewing timeT, sorted by coherences. DD model with variable drift: scale coher. expt. asmpt. decay rate We can predict Psychometric Functions (PMFs): Accuracy Model fits to data from monkeys during training on the moving dots task. Animals learn to extract signal from noise. Scale factor a: steady increase in SNR. (Eckhoff, H, Law, Conolly, Gold, New J. Phys., 2008)

26. Perturbing a nonlinear integrator A nonlinear model can also be distinguished from CD, TD, SOU and UOU: both mean and SD effects decrease monotonically as T increases. The “simple” patterns seen here occur only for low noise and early perturbations (T << mean RT). Experiments are in planning stage (Usher, McClelland, Simen). Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.