Motivation and Cognition: From Regulatory Fit to Reinforcement Learning

1. Motivation and Cognition: From Regulatory Fit to Reinforcement Learning Darrell A. Worthy University of Texas, Austin

2. Motivation and Cognition • Why study motivation? • Need to understand how goals and rewards influence cognition and behavior. • More complicated than anecdotal notions • Approach vs. avoidance distinction • Global incentive vs. local goal pursuit mechanism • Leads to regulatory fit or mismatch • Regulatory fit affects cognition and behavior

3. Overview of Talk • Regulatory Fit Framework • Regulatory fit affects cognition • Tests of the Regulatory Fit Hypothesis • Extend framework to examine effects of social pressure. • Regulatory Fit and Decision-making • Future Directions

4. Regulatory Fit Framework Global Incentive Promotion Focus Prevention Focus • Global incentive focus interacts with local reward structure • Produces a Fit or a Mismatch (e.g. Higgins, 2000). • Almost all cognitive research involves promotion focus with gains reward structure. Gains Local Goal Pursuit Mechanism Losses

5. Manipulation of Regulatory Focus(Global Task Goal)

6. Manipulation of Goal-Pursuit Mechanism(Local Trial-by-trial Task Goal)

7. Effects of Regulatory Fit • Previous research • Regulatory Fit leads to: • Increased sense of ‘feeling right’ (Higgins, 2000) • Increased motivational strength (Spiegel et al., 2004) • Increased “cognitive flexibility” (Shah et al., 1998) • Flexibility can be defined within tasks • Category-learning -willingness to test various strategies • Decision-making -willingness to explore the environment

8. Perceptual Classification • Excellent for testing the effects of regulatory fit • Stimuli with small number of dimensions • Lines that vary in length, orientation and position • ‘Gabor’ patches that vary in frequency and orientation • Experimenter control of category structure • Extensive set of tools for modeling performance of individual participants • Can assess the strategies participants use in the task

9. Explicit, Hypothesis-testing system mediates learning of “rule-based” (RB) category structures. -Frontally mediated -Verbalizable rules Implicit, Procedural learning system mediates learning of “information-integration” (II) category structures. -Striatally mediated - Verbalizable rules hurt performance (Maddox and Ashby, 2004; Ashby et al., 1998) Multiple systems mediate different classification tasks

10. Categorization Tasks Rule-Based Information-Integration Learned Explicitly Learned Implicitly

11. Increased cognitive flexibility will increase rule use Enhance performance on rule-based tasks Will harm performance on information-integration task Rule-use disrupts the procedural system Recent tests of this hypothesis (Markman et al., 2005; Maddox et al., 2006; Grimm et al., 2008) Manipulated regulatory focus and reward structure between subjects Used rule-based and information-integration tasks Influence of Regulatory Fit

12. Regulatory Fit and Classification • Rule-based performance was better in a fit • Information-integration performance was better in a mismatch • Fit increases rule-use • Helps on rule-based, hurts on information-integration

13. Choking & Excelling Under Pressure Worthy, Markman, & Maddox, 2009a, 2009b; Worthy, Markman, & Maddox, 2008; Markman, Maddox & Worthy 2006

14. Choking Under Pressure • Anecdotal phenomenon (e.g. sports, test-taking, etc.) • People perform worse than normal when under pressure • Some also seem to excel under pressure • Might pressure be similar to a prevention focus?

15. Motivation and Pressure • Working Memory Distraction Hypothesis of choking • Pressure reduces WM capacity • Should see main effects • Pressure decreases rule-use • Alternative: Pressure affects cognition through its effects on the motivational state • Working Hypothesis: • Pressure induces an “avoidance” or “prevention” motivational state • Interacts with goal pursuit mechanism to influence regulatory fit

16. Pressure and Category-Learning Global Incentive • Low pressure – “do your best” • High pressure: -Paired with a ‘partner’ -If both of you reach criterion, both get $6 -If one of you fails neither get $6 bonus -Partner has already reached criterion -Trying to prevent the negative end-state of letting their partner down • Run gains and losses Promotion Focus Prevention Focus Low Pressure High Pressure Gains Local Goal Pursuit Mechanism Losses

17. WM Distraction vs. Regulatory Fit • Pressure decreases WM • Poor rule-based performance • Better information-integration • Pressure induces a prevention focus • Will interact with the reward structure

18. Method • 2 (Pressure-level) X 2 (Reward Structure) X 2 (Task Type) between-subjects design • Performed 8 80-trial blocks Rule-Based Information-Integration Worthy, et al., 2009, Psychonomic Bulletin & Review

19. Results Worthy, et al., 2009, Psychonomic Bulletin and Review

20. Decision Bound Modeling Used to infer strategy use. Decision bound models assume stimuli are classified based on which side of the decision bound they fall on Several models are fit to the data Best-fitting model gives information about which strategy each participant probably used to classify the stimuli

21. Decision Bound Modeling

22. Model Fitting Procedure Fit each participant’s data on a block-by-block basis Used AIC to determine best fitting model for that block Penalizes for free parameters Examined the proportion of data sets best fit by each model over all blocks of the task.

23. Model-Based Analysis • Best strategy for rule-based task • Best strategy for information-integration task

24. Proportion Fit by Best Model • Increase in accuracy likely due to improved strategy use. Worthy, et al., 2009, Psychonomic Bulletin & Review

25. Summary • Pressure does appear to operate like a prevention focus during classification learning. • Not main effect where WM is decreased • Gains mismatches with pressure-induced prevention focus • Pressure hurts rule-based performance • Pressure helps information-integration performance. • Losses fits with pressure-induced prevention focus • Pressure helps rule-based performance. • Pressure hurts information-integration performance.

26. Pressure and Experts • Examined effects of pressure after extensive training. • RB or II task • 5 640-trial sessions Supports a different account for effects of pressure on experts Worthy et al., 2009, Attention, Perception and Psychophysics

27. Real World Choking • Examined clutch free-throw performance among NBA athletes • Considered point-differential between shooter’s team. • Compared percentage to career percentage Worthy et al., 2009, International Journal of Creativity and Problem Solving

28. Regulatory Fit and Decision-Making Worthy, Maddox, & Markman, 2007

29. Decision-making from experience • Basic Design • ‘Gambling’ task • Participants choose from two or more decks of cards • Must either maximize gains or minimize losses Gains Losses

30. Modeling • Task is amenable to reinforcement learning modeling • Can estimate parameters that describe performance

31. Expected Value (EV) • EV – How many points one expects to gain or lose from selecting a given deck • Used to determine which option to choose • Example • EVred deck= 7 points • EVblue deck= 3 points

32. Exploration/Exploitation Dilemma • Exploit the option with the highest EV or • Explore other options with lower EVs • Must balance the need to exploit with the need for new information • Exploration may be more frontally mediated (e.g. Daw et al., 2006). • Working hypothesis: Regulatory fit will increase exploration

33. Task Design • Can design tasks to favor more exploratory or exploitative strategies. • Experiment 1 – Exploration-optimal • Experiment 2 – Exploitation-optimal (Gains only) • Use behavioral and model-based analyses to test the regulatory fit hypothesis Worthy et al., 2007

34. Experiment 1 • Designed a task where exploring the deck with lower EV led to better-long-term performance. • Had to be willing to explore the Advantageous deck • Fit should increase exploration; performance

35. Methods • Used raffle-ticket manipulation to manipulate regulatory focus

36. Methods • Achieved global criterion by either maximize gains or minimizing losses

37. Behavioral results • Participants in a regulatory fit came significantly closer to the performance criterion than participants in a mismatch

38. Modeling Choice Behavior • EVs of each option are updated via an exponential recency-weighted algorithm New EV Current EV Recency Parameter Reward Current EV • If reward is greater than the current EV the EV increases • If reward is less than the current EV the EV decreases

39. Action Selection Action selection is probabilistically determined via choice rules (e.g. Luce, 1959) Softmax Rule Exploitation parameter EV for option “A” Probability of choosing option “A” Sum of EVs for all options • Higher g values indicate greater exploitation • Lower g values indicate greater exploration • Can directly parameterize degree of exploratory vs. exploitative behavior

40. Model-based results • Fit reinforcement-learning model to estimate the degree of exploratory vs. exploitative behavior. • Participants in a regulatory fit had significantly lower estimated exploitation-parameter values.

41. Experiment 2 • Designed a task where exploitation of the deck with the best expected value led to the best performance. • If fit increases exploration then participants in a fit should do worse.

42. Results • Only ran participants with a gains reward structure • Participants in a regulatory fit were further from the performance criterion

43. Model-Based Results • Participants in a fit were less exploitative than those in a mismatch

44. Summary • Regulatory fit influenced the decision-making behavior • Fit – greater exploration • Mismatch greater exploitation • Social pressure induces a prevention focus • Influences regulatory fit • Differential performance on category-learning tasks • Three-way interaction • Regulatory focus – Promotion vs. prevention • Reward Structure – Maximize gains vs. minimize losses • Task Demands – Rule-based vs. information-integration; exploration-optimal vs. exploitation-optimal

45. Expected Reward Comparison • Extended decision-making paradigm to ratio vs. difference comparisons • Are EVs compared via ratio or differences? • Manipulated whether difference or ratio preserved. • Changing the ratio between EVs affected performance Worthy et al., 2008, Memory and Cognition

46. Research Approach • Categorization and Decision-making tasks • Behavioral analysis • Mathematical modeling • Decision-bound modeling • Reinforcement-learning modeling • Ground theories in neuroscience • Leads to novel predictions

47. Current & Future Directions • ‘Why’ does regulatory fit influence behavior and cognition • Working memory hypothesis • Fit increases WM memory capacity • Not yet directly tested • Test using regulatory focus and social pressure manipulation in WM tasks. • Test by adding WM span as an additional factor on categorization and decision-making tasks. • Regulatory fit and short-term vs. long-term decision-making • Does fit reduce future discounting? • People in a fit may focus more on long-term outcomes

48. Current & Future Directions • Individual Differences • Are some less susceptible to situational factors than others? • Why do some people tend to choke, while others excel? • Aging and decision-making • Older adults appear to be more exploratory than younger adult • May value long-term over short-term outcomes • Positivity bias • Neural differences • Gender and decision-making • Men appear to be more exploitative than women

49. Current & Future Directions • Social vs. Monetary rewards • Give incrementally happier or angrier faces as feedback in decision-making tasks. • Can use same modeling approach • Compare to monetary rewards • Neural mechanisms

50. Current & Future Directions • Category learning • Feedback timing • Very important for procedural learning system • Retention and generalization • Desirable difficulties • Naturalistic stimuli (x-rays – tumor detection) • Interactions between multiple systems – competition vs. cooperation