The Speed-Accuracy Trade-Off in Decision-Making: Theory, Behavior, and Neural Bases

1. Régulation du compromis vitesse-précision pendant la prise de décision et l’action théorie, comportement et bases neuronales David Thura Département de neurosciences Universitéde Montréal Université du Québec à Montréal 18 janvier 2018

2. What usually motivates animals to behave ? • Get high value rewards (e.g. food) • Avoid punishments (errors) and effort costs How to do this? • Interactive behavior: successive decisions and actions • Prioritize performance (highest possible accuracy) ? No, animals do not usually behave “optimally”, in terms of performance (e.g. Balci et al., 2011; Chittka et al., 2009) • Because high accuracyis time consuming

3. An old (and trivial) observation: Accuracy varies with response time

4. Accuracy varies with response time For a given task difficulty • Inverse relationship between response time and outcome accuracy • True during decision-making and movement execution (Fitts’s law) • Ubiquitous in human and animals, from ants to monkeys • Robust across task domains Time cost high Accuracy gain Accuracy Speed gain Accuracy cost low low high Response time (speed) • Accuracy cost : obvious, leads to inaccurate decision and potential punishment • Time cost: elapsing time increases the risk of missing the goal (deadline). It reduces the value of reward (temporal discounting) as well as chances for other opportunities (reward rate)

5. An example of SAT in the wild Where is the fly? Hoverflies use mimicry (Batesian mimicry) to force predator (birds, bats, frogs, spiders) to increase their decision accuracy, giving them sufficient time to escape Chittka et al. (2009) Science

6. Instead of performance, animals are instead motivated to find the good balance between speed and accuracy = Speed-accuracy trade-off (SAT) In the wild, animals usually behave to optimize their reward rate (e.g. Balci et al., 2011; Chittka et al., 2009) = Reward rate • properly set the speed/accuracy trade-offduring decision-making and movement execution To optimize the reward rate one needs to: Context dependent (high time pressure, high error cost,…) • efficiently adjust the speed-accuracy trade-off when the context changes

7. Investigation of SAT • The covariation between response speed and accuracy during decision-making is seen as a signature of the decision process itself. • Consequently, experimental investigations of SAT progressed largely in parallel with mathematical models of the decision process.

8. Theoretical models of decision-making “Drift diffusion” or “bounded integrator” models (Ratcliff, 1978) E ∫ D Sensory evidence decision Neural activity builds to a threshold by accumulatingsensory evidence over time

9. Theoretical models of decision-making The drift diffusion framework (For application to SAT investigation, see Bogacz et al., 2010) Option A Decision threshold for A accuracy Distance to travel (excursion) Activation speed Decision variable error Decision threshold for B Option B Time

10. In the brain, the “threshold” does not vary with reaction time Movement selection (decision-making) Movement initiation Roitman & Schadlen (2002) J Neurosci Hanes & Schall (1996) Science

11. Theoretical models of decision-making The drift diffusion framework (For application to SAT investigation, see Bogacz et al., 2010) Adjustment of baseline Equivalent to Speed context Accuracy context Time Time

12. Theoretical models of decision-making Urgency-gating model Cisek et al. (2009) J Neurosci Evidence-independent signal = Gain DV * gain with constant threshold Equivalent to SI D x U Speed context Gain Accuracy context Time Time Time Urgency Time Time

13. Why do we think the diffusion model is not appropriate for natural decisions? Environment is constantly and unpredictably changing during interactive behavior. One needs to react quickly. Integrators are too slow to process these changes in real-time. What happens if no more information ever comes? Something is needed to push the system to act anyway. Time-related information is crucial: Need to properly adjust the speed/accuracy trade-off (SAT)

14. Dynamic decisions by urgency-gating (Cisek et al., JN, 2009; Thura et al., JNP, 2012) Environment Brain Sensory processing Integration of new evidence Decision threshold Sensory information Action execution E Deliberation and commitment C x Elapsing time U + Context Urgency signal

15. Dynamic decisions by urgency-gating Thura et al. (2012) J Neurophysiol Theoretical demonstration of optimal behavior evidence favors A Only novel information (and not the entire state) needs to be integrated (= low-pass filtered sensory evidence). A time-dependent gain (the urgency) pushes the system to act. It implements a dropping threshold. Urgency setting controls the SAT E evidence favors B E criterion for choosing A DDM ∫E F[E] UGM criterion for choosing B U The urgency-gating model responds to changes quickly and implements the SAT during decision-making The model explains behavioral data of human subjects making decisions in noisy and changing conditions.

16. Aim Exploration of the computations and neural substrates of SAT setting and adjustments blocks of trials within between

17. The tokens task Sedna & Zola

18. The tokens task Sedna & Zola

19. Successprobability 1 Time mRT Probablility ofcorrect choice 0.5 Movement onset Decision time Target reached 0 0 500 1000 1500ms time The tokens task “deliberation” “commitment” “SAT” • Reaction time task • Changing evidence • Discriminates accumulation from urgency-gating models • Dissociates deliberation from commitment • Encourages SAT

20. Classifying trial types 1.0 easy 0.8 ambiguous 0.6 Success probability 0.4 misleading 0.2 0.0 Time

21. Behavioral data 15 • In easy trials, monkeys respond more quickly than in ambiguous or misleadingtrials • Late decisions are made at a lower level of success probability • Monkeys drop their accuracy criterion as time is passing • Urgency increases over time 1.4 100 10 1.2 Percentage of trials 1 Accuracy criterion 5 0.8 50 Sensory evidence at decision (SumLogLR) urgency 0 0.6 0 1.0 2.0 3.0 Decision duration (s) 0.4 0 0 0.5 1 0.2 Success probability 0.1 0.5 0.9 1.3 1.5 1.9 2.3 2.7 0 Decision duration (s) Thura & Cisek (2014) Neuron

22. Neural recordings • Sensorimotor network • Involved in reach preparation and execution • Also involved in action selection (decision-making)? • Involved in SAT adjustments? M1 PMd eye

23. Decision-making about actions • Defined by the environment during interactive behavior • Fast • Dynamic • Updated in real-time • The brain has been “elaborated” and conserved through evolution to deal with these concrete and pragmatic decisions The affordance competition hypothesis (Cisek, 2007)

24. Directional tuning – Preparatory activity Cell’s non-preferred target Cell’s preferred target

25. Neural activity in PMd and M1 Thura & Cisek (2014) Neuron Preferred target Opposite target Dorsal premotor cortex • During deliberation • “Decision-related cells” • Continuously reflect the profile of evidence for their preferred target • Also build-up over time Primary motor cortex • During deliberation • Similar results 0.5 Evidence 30 25 PMd, n=124 (45%) 20 Mean neural response (Hz) 15 10 5 40 M1, n=137 (68%) 30 Mean neural response (Hz) 20 0 0 0.5 0.5 1.0 1.0 1.5 1.5 10 Time from first token jump (s) Time from first token jump (s)

26. Relationship between sensory evidence and neural activity PMd (n=124) M1 (n=137) Token jump #1 Jump #2 Jump #3 Jump #4 Jump #5 Jump #6 Jump #7 Jump #8 Jump #9 8 6 Normalized activity 4 2 0 -1 0 1 -1 0 1 -2 0 2 -2 0 2 -2 0 2 -2 0 2 -2 0 2 -2 0 2 -2 0 2 Evidence to cells’ PT (SumLogLR unit) • PMd and M1: Strong monotonic relationship to evidence

27. Neural activity in PMd and M1 Thura & Cisek (2014) Neuron Preferred target Opposite target Dorsal premotor cortex • Close to movement • Reach a peak about 280ms before movement onset • Critical contrast reached • “Moment of commitment” Primary motor cortex • Close to movement • Peak is ~140ms later • Deliberation and commitment in PMd/M1 0.5 Evidence 30 25 PMd, n=124 (45%) 20 Mean neural response (Hz) 15 10 5 40 M1, n=137 (68%) 30 Mean neural response (Hz) 20 0 0 0.5 0.5 1.0 1.0 1.5 1.5 -1.5 -1.5 -1.0 -1.0 -0.5 -0.5 0 0 0.5 0.5 10 Time from first token jump (s) Time from reach onset (s)

28. PMd, n=124 30 Compare to activity in PFC 25 20 15 30 M1, n=137 20 Mean neural response (Hz) Prefrontal cortex • During deliberation • Strong influence of evolving sensory evidence • Close to movement • No consistent peak of activity around the “moment of commitment” Provides the sensory information but not involved in commitment PFC, n=13 30 -2.0 -1.0 0 1.0 25 Time from reach onset (s) 20 Mean neural response (Hz) 15 10 5 0 0.5 1.0 1.5 Time from first token jump (s)

29. Interim Sensorimotor reach areas : Decision determination Competition Actions Evidence PFC Premotor Motor

30. Speed-accuracytrade-off manipulation Slow blocks : encourage slow and accurate decisions Fastblocks : encourage fast and risky decisions = SAT context No acceleration : wait until all tokens have jumped Commitment time Fastblock Slowblock 1 RT Movement Time saved Higher reward rate Probability of correct choice 0.5 Reward Reward No post-decision interval: random guesses 0 0 500 1000 1500ms time

31. Example session (Monkey Z) 2.0 • Monkeys make faster decisions in the fast blocks • In the fast blocks, the probability of success decreases • The rate of reward significantly increases in the fast regime • Monkeys trade speed for accuracy between the blocks to increase their reward rate Slow blocks DTs (s) Fast blocks 1.0 0 1 0.8 Success probability 0.6 0.4 3 2 Reward rate (drop/s) 1 0 0 100 200 300 400 500 600 Trials Thura et al. (2014) J Neurosci

32. Urgency influences decision-making and action execution Urgency-gating model Cisek et al. (2009) J Neurosci SI D 1.4 x Slow 1.2 U 1 • Less sensory information is needed as time is passing • Because of the urgency to commit • Urgency is adjusted between the blocks • Urgency also influences movements 0.8 Sensory evidence at decision Fast 0.6 decision Sensory info. Urgency 0.4 Data Best fit 2 0.2 0 0.1 0.9 1.7 2.5 1.5 Decision time (s) Urgency 1 240 0.5 0 220 0 1.0 2.0 3.0 Time (s) Reach peak velocity (cm/s) action 200 • A common mechanism for SAT adjustment during decision-making and movement execution 180 0.1 0.9 1.7 2.5 Decision time (s) Thura et al. (2014) J Neurosci

33. Neural recordings Slow versus fast blocks M1 PMd PFC eye

34. Decision-related neural activity in PMd and M1 is amplified in the fast blocks Easy trials Misleading trials Slow blocks 30 Fast blocks 25 20 PFC Neural activity (Hz) 15 10 13 cells 13 cells 5 0 0.5 1.0 1.5 0 0.5 1.0 1.5 30 • In the fast block, activity is slightly amplified in PFC • The effect of block is stronger in PMd/M1 • Is this amplification due to urgency? 25 20 PMd Neural activity (Hz) 15 10 58 cells 45 cells 5 0 0.5 1.0 1.5 0 0.5 1.0 1.5 30 25 20 M1 Neural activity (Hz) 15 10 59 cells 74 cells Thura & Cisek (2016) J Neurosci 0 0.5 1.0 1.5 0 0.5 1.0 1.5 Time from trial start (s) Time from trial start (s)

35. Urgency modulates activity in the sensorimotor cortex Thura & Cisek (2016) J Neurosci PMd M1 PMd M1 Slow blocks Fast blocks 35 28 30 Token jump 25 24 #1 decision Activity (Hz) #3 Average FR at 0 evidence (Hz) 20 20 #5 15 #7 #9 16 10 #11 2 5 12 -2 -1 0 1 2 -2 -1 0 1 2 0.1 0.9 1.7 0.1 0.9 1.7 Evidence (SumLogLR) Evidence (SumLogLR) Time (s) Time (s) 1.5 Urgency 1 PMd **** 0.5 30 26 0 25 0 Speed-related activity (Hz) 20 22 action 15 *** 18 10 **** n = 9 5 14 -0.6 -0.4 -0.2 0 0.2 0.5 1.0 1.5 2.0 Time from movement (s) Decision duration (s) 1.0 2.0 3.0 Time (s) 35

36. decision dlPFC PMd M1 sensory evidence action Urgency Proposed role of urgency during action selection and execution Control the speed of decisions (setting of the SAT) and invigorate actions

37. Origin of the urgency signal? The affordance competition hypothesis (Cisek, 2007)

38. The BG: Origin of the urgency signal? Decision Action Urgency • Anatomy and connectivity • Loops between BG and cortex • regulate cortical activity • Functions • - SAT regulation • Bogacz et al. 2010 • Forstmann et al. 2010 • - Vigor of movements • Mazzoni et al. 2007 • Turner and Desmurget, 2010 PMd M1 ACC lPFC OFC Th. vmPFC Basal ganglia

39. PMd, n=124 30 Activity in the globus pallidus 25 20 15 • During deliberation • GPe: weaker relation to sensory evidence • GPi: almost no relation Dorsal premotor cortex • During deliberation • “Decision-related cells” • Continuously reflect the profile of evidence for their preferred target • Also build-up over time Primary motor cortex • During deliberation • Similar results 30 M1, n=137 20 GPe, n=29 90 Mean neural response (Hz) 80 70 60 GPi, n=29 100 Globus pallidus 80 Mean neural response (Hz) Thura & Cisek (2017) Neuron 0 0.5 1.0 1.5 Time from first token jump (s)

40. Pallidal “ramping” activity Thura & Cisek (2017) Neuron GPe Build-up cells (n=19) Decreasing cells (n=5) 90 • Some cells show “ramping” • Build-up cells • Excite cortex? • Decreasing cells • Inhibit cortex? • Reflect speed-accuracy trade-off? • Build-up cells are more active in the “fast” block • Decreasing cells are less active in the “fast” block • Even during baseline • Consistent with “urgency” signal 80 Mean (± CI) neural response (Hz) 70 Slow Block Fast 60 -0.5 0 0.5 1.0 1.5 -0.5 0 0.5 1.0 1.5 Time from first token jump (s) Time from first token jump (s) GPi Build-up cells (n=11) Decreasing cells (n=11) 100 90 Mean (± CI) neural response (Hz) 80 70 60 -0.5 0 0.5 1.0 1.5 -0.5 0 0.5 1.0 1.5 Time from first token jump (s) Time from first token jump (s)

41. Local field potentials in the GPi Single session Involved in sensory-motor processes, action preparation and/or arousal Power β oscillations Slow Fast -3 10 0.1 -20 baseline deliberation 60 baseline deliberation 3 0 -30 40 2 Frequency (Hz) -0.1 Voltage -40 Power 20 1 -50 -0.2 -4 -3 10 10 0 0 -60 -0.3 -500 0 500 1000 -500 0 500 1000 0 500 1000 1500 20 40 60 80 100 Time from first token jump (ms) Time (ms) Frequency (Hz) Average spectrum across 17 sessions Low beta (17-22Hz) High beta (27-30Hz) Slow 1.5 Block Fast 7 6 1 Power 5 Power 4 0.5 3 2 0 1 0 20 40 60 0 500 1000 0 500 1000 Frequency (Hz) Time from first token jump (ms) Thura & Cisek, In preparation

42. Another role for the globus pallidus PMd, n=124 30 25 20 15 Dorsal premotor cortex • Close to movement • Reach a peak about 280ms before movement onset • Critical contrast reached • “Moment of commitment” Primary motor cortex • Close to movement • Peak is ~140ms later • Close to movement • Both GPe and GPi become tuned just around the “moment of commitment” • Not deliberation, nor commitment but confirmation of the choice 30 M1, n=137 20 GPe, n=29 (57%) 90 Mean neural response (Hz) 80 70 60 GPi, n=29 (52%) 100 Globus pallidus 80 Mean neural response (Hz) Thura & Cisek (2017) Neuron 0 0.5 1.0 1.5 -1.5 -1.0 -0.5 0 0.5 Time from first token jump (s) Time from reach onset (s)

43. Summary Dynamic decision about actions Within a given behavioral repertoire (two reaching movements) No learning • Determined in the sensorimotor cortex • The prefrontal cortex provides the sensory evidence, but does not determine the commitment • The basal ganglia do not influence deliberation but confirm commitment • The BG provide the urgency signal that controls the SAT during decision-making and action execution actions evidence urgency M1 PMd competition PPC dlPFC Global SAT signal GPe GPi

44. Acknowledgments http://davidthura25.wix.com/neurophylab Paul CISEK Former colleagues / students Ignasi Cos Jessica Trung Charles-William Fradet Julie Beauregard-Racine Guido Guberman Matt Carland Ayuno Nakahashi Julien Michalski Marie-Claude Labonté Sedna & Zola

45. Interim Sensorimotor reach areas : Decision determination, global SAT adjustments Competition Actions Evidence urgency PFC Global SAT signal Premotor Motor GPe GPi

46. Trial to trial adjustment of SAT Time Guido Guberman Trial n Trial n+1 Decision Outcome Reaction time & success probability Correct Deliberation C Error Neural activity in PMd / M1

47. Post-error adjustments - Behavior After a correct choice After an error Monkey S • In the fast blocks, both monkeys made slower decisions following an error. • This phenomenon (post-error slowing) has been reported in many species and in various tasks • These longest decisions are made with at a slightly higher SP  local SAT • In the slow blocks, one monkey showed some post-error speeding 2000 0.78 1800 Decision duration (ms) Success probability 1600 0.74 1400 0.7 1200 Slow Fast Slow Fast Monkey Z 1800 0.8 1600 0.75 Decision duration (ms) Success probability 1400 0.7 1200 1000 0.65 Slow Fast Slow Fast Thura, Guberman & Cisek(2017) J Neurophysiol

48. Effect of previous trial outcome on PMd activity Slow block Fastblock Slow block After correct After error 30 30 commitment 25 25 20 20 Activity (Hz) 15 Activity (Hz) 15 10 10 5 5 Cell #282 0 0 -0.5 0 0.5 1.0 1.5 2.0 -0.5 0 0.5 1.0 1.5 -0.5 0 0.5 1.0 1.5 Time from first token jump (s) Time from first token jump (s) Time from first token jump (s) 70 80 60 50 60 40 Activity (Hz) Activity (Hz) 40 30 20 20 10 Cell #285 0 0 -0.5 0 0.5 1.0 1.5 2.0 -0.5 0 0.5 1.0 1.5 -0.5 0 0.5 1.0 1.5 Time from first token jump (s) Time from first token jump (s) Time from first token jump (s)

49. Previous trial outcome modulates PMd activity Slow blocks Fastblocks Monkey S Thura, Guberman & Cisek(2017) J Neurophysiol 60 18*/154 23*/154 • Out of 235 PMd cells, we found that 54 (23%) were modulated by the previous trial outcome • Some cells were more active after a correct choice (C-cells), some after an error (E-cells) • Slightly more cells modulated in the fast blocks compared to the slow blocks • Some cells were modulated in the two conditions. In that case, modulation was always in the some direction 50 40 Baseline activity (Hz) [AE] 30 20 26% 32% 10 22*/154 26*/154 0 0 20 40 60 0 20 40 60 Baseline activity (Hz) [AC] Baseline activity (Hz) [AC] Monkey Z 50 15*/81 10*/81 40 30 Baseline activity (Hz) [AE] 20 20% 28% 10 11*/81 13*/81 0 0 10 20 30 40 50 0 10 20 30 40 50 Baseline activity (Hz) [AC] Baseline activity (Hz) [AC]

50. Averaged responses of C and E cells Thura, Guberman & Cisek(2017) J Neurophysiol Cells modulated in the fast blocks • Activity of C-cells strongly builds-up during deliberation in all conditions. By contrast, activity of E-cells appear more stable during deliberation • The effect of previous trial outcome seems to vanish more quickly during deliberation for the E-cells than for the C-cells • Activity of C-cells reaches a fixed peak close to commitment time. Much less fixed in E-cells • C-cells, but not E-cells, are involved in the decision process 25 20 20 15 Activity (Hz) 15 After correct After error 10 10 39 C-cells 5 5 -1.5 -1.0 -0.5 0 0.5 -0.5 0 0.5 1.0 1.5 2.0 25 20 20 15 15 Activity (Hz) 10 10 33 E-cells 5 5 -1.5 -1.0 -0.5 0 0.5 -0.5 0 0.5 1.0 1.5 2.0 Time from movement onset (s) Time from first token jump (s)