Model-based RL (+ action sequences): maybe it can explain everything

1. Model-based RL (+ action sequences): maybe it can explain everything Niv lab meeting 6/11/2012 Stephanie Chan

2. goal-directed v.s. habitual instrumental actions • Habitual • Goal-directed • After extensive training • Choose action based on previous actions/stimuli • Sensory motor cortices + DLS (putamen) • Not sensitive to: • reinforcer devaluation • action-outcome changes in contingency • After moderate training • Choose action based on expected outcome • PFC & DMS(caudate) • Usually: • Model-based RL • Model-free RL

3. goal-directed v.s. habitual instrumental actions • What do real animals do?

4. Model-free RL • Explains resistance to devaluation: • Devaluation occurs in “extinction”. No feedback / no TD error • Does NOT explain resistance to changes in action-outcome contingency • In fact, habituated behavior should be MORE sensitive to changes in contingency • Maybe: update rates go small after extended training

5. Alternative explanation • We don’t need model-free RL • Habit formation = association of individual actions into “action sequences” • More parsimonious • A means of modeling action sequences

6. Over the course of training • Exploration -> exploitation • Variability -> stereotypy • Errors and RT -> decrease • Individual actions -> “chunked” sequences • PFC + associative striatum -> sensorimotor striatum • “closed loop” -> “open loop”

7. When should actions get chunked? • Q-learning with dwell time • Q(s,a) = R(s) + E[V(s’)] – D(s)<R> • When costs (possible mistakes) are outweighed by benefits (decrease decision time) • Cost: C(s,a,a’) = E[Q(s’,a’)-V(s’)] = E[A(s’,a’)] • Efficient way to compute this: TDt = [rt – dt<R> + V(st+1)]-V(st) = a sample of A(st,at) • Benefit: (# timesteps saved) <R>

8. When do they get unchunked? • C(s,a,a’) is insensitive to changes in environment • Primitive actions no longer evaluated, no TD error, no samples for C • But <R> is sensitive to changes… • Action sequences get unchunked when environment changes to decrease <R> • No unchunking if environment changes to present a better alternative to increase <R> • Ostlund et al 2009: rats are immediately sensitive to devaluation of the state that the macro action lands on, but not on the intermediate states

9. Simulations I : SRTT-like task

10. Simulations II: Instrumental conditioning Reinforcer devaluation Non-contingent Omission