CS 5368: Artificial Intelligence Fall 2010

1. CS 5368: Artificial IntelligenceFall 2010 Lecture 12: MDP + RL (Part 2) 10/14/2010 Mohan Sridharan Slides adapted from Dan Klein

2. Recap: Reinforcement Learning • Basic idea: • Receive feedback in the form of rewards. • Agent’s utility is defined by the reward function. • Must learn to act so as to maximize expected rewards.

3. Reinforcement Learning Problem: Find values for fixed policy  (policy evaluation): Model-based learning: Learn the model, solve for values. Model-free learning: Solve for values directly (by sampling).

4. The Story So Far: MDPs and RL Techniques: Things we know how to do: • If we know the MDP • Compute V*, Q*, * exactly. • Evaluate a fixed policy . • If we don’t know the MDP: • We can estimate the MDP then solve. • We can estimate V for a fixed policy . • We can estimate Q*(s,a) for the optimal policy while executing an exploration policy. • Model-based RL: • Value and policy Iteration. • Policy evaluation.. • Model-free RL: • Value learning. • Q-learning.

5. Passive Learning • Simplified task: • You do not know the transitions T(s,a,s’). • You do not know the rewards R(s,a,s’). • You are given a policy (s). • Goal: learn the state values (and the model?). Policy evaluation. • In this case: • Learner “along for the ride”. • No choice about what actions to take. Just execute the policy and learn from experience. • This is NOT offline planning!

6. Active Learning • More complex task. • R and T still unknown. Does not have fixed policy that determines its behavior! • Must learn what actions to take. • Same set of algorithms can be modified to address the challenges. • We begin with passive model-based and model-free methods.

7. Example: Direct Estimation y • Episodes: +100 (1,1) up -1 (1,2) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (3,3) right -1 (4,3) exit +100 (done) (1,1) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (4,2) exit -100 (done) -100 x  = 1, R = -1 V(2,3) ~ (96 + -103) / 2 = -3.5 V(3,3) ~ (99 + 97 + -102) / 3 = 31.3

8. Model-Based Learning • Idea: • Learn the model empirically through experience. • Solve for values as if the learned model were correct. • Simple empirical model learning: • Count outcomes for each s, a. • Normalize to give estimate of T(s, a, s’). • Discover R(s, a, s’) when we experience (s, a, s’). • Solving the MDP with the learned model: • Iterative policy evaluation, for example: s (s) s, (s) s, (s),s’ s’

9. Example: Model-Based Learning y • Episodes: +100 (1,1) up -1 (1,2) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (3,3) right -1 (4,3) exit +100 (done) (1,1) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (4,2) exit -100 (done) -100 x  = 1 T(<3,3>, right, <4,3>) = 1 / 3 T(<2,3>, right, <3,3>) = 2 / 2

10. Model-Free Learning • Want to compute an expectation weighted by P(x): • Model-based: estimate P(x) from samples, compute expectation. • Model-free: estimate expectation directly from samples. • Why does this work? Because samples appear with the right frequencies!

11. Sample-Based Policy Evaluation? • Who needs T and R? Approximate the expectation with samples (drawn from T). s (s) s, (s),s’ s, (s) s’ s1’ s3’ s2’ Almost! But we only actually make progress when we move to i+1.

12. Temporal-Difference Learning • Big idea: learn from every experience! • Update V(s) each time we experience (s,a,s’,r) • Likely s’ will contribute to updates more often. • Temporal difference learning: • Policy can still be fixed! • Move values toward value of whatever successor occurs: running average! s (s) s, (s) s’ Sample of V(s): Update to V(s): Same update:

13. Example: TD Policy Evaluation (1,1) up -1 (1,2) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (3,3) right -1 (4,3) exit +100 (done) (1,1) up -1 (1,2) up -1 (1,3) right -1 (2,3) right -1 (3,3) right -1 (3,2) up -1 (4,2) exit -100 (done) Take  = 1,  = 0.5

14. s a s, a s,a,s’ s’ Problems with TD Value Learning • TD value leaning is a model-free way to do policy evaluation. • However, if we want to turn values into a (new) policy, we are sunk: • Idea: learn Q-values directly. • Makes action selection model-free too!

15. Return to Active Learning • Full reinforcement learning • You do not know the transitions T(s,a,s’) • You do not know the rewards R(s,a,s’) • You can choose any actions you like. • Goal: learn the optimal policy. • … what value iteration did! • In this case: • Learner makes choices! • Fundamental tradeoff: exploration vs. exploitation. • You actually take actions in the world and find out what happens.

16. Model-Based Active Learning • In general, want to learn the optimal policy, not evaluate a fixed policy. • Idea: adaptive dynamic programming  • Learn an initial model of the environment. • Solve for the optimal policy for this model (value or policy iteration). • Refine model through experience and repeat. • Crucial: we have to make sure we actually learn about all of the model.

17. Example: Greedy ADP • Imagine we find the lower path to the good exit first. • Some states will never be visited following this policy from (1,1). • Can keep re-using this policy but following it never explores the regions of the model we need in order to learn the optimal policy . ? ?

18. What Went Wrong? • Problem with following optimal policy for current model: • Never learns about better regions of the space if current policy neglects them. • Fundamental tradeoff: exploration vs. exploitation. • Exploration: take actions with suboptimal estimates to discover new rewards and increase eventual utility. • Exploitation: once the true optimal policy is learned, exploration reduces utility. • Systems must explore in the beginning and exploit in the limit. ? ?

19. Detour: Q-Value Iteration • Value iteration: find successive approx optimal values • Start with V0*(s) = 0, which we know is right (why?) • Given Vi*, calculate the values for all states for depth i+1: • But Q-values are more useful! • Start with Q0*(s,a) = 0, which we know is right (why?) • Given Qi*, calculate the q-values for all q-states for depth i+1:

20. Q-Learning • We would like to do Q-value updates to each Q-state: • But cannot compute this update without knowing T, R. • Instead, compute average as we go: • Receive a sample transition (s,a,r,s’). • This sample suggests: • But we want to average over results from (s,a) (Why?) • So keep a running average:

21. Q-Learning Properties • Will converge to optimal policy: • If you explore enough (i.e. visit each q-state many times). • If you make the learning rate small enough. • Basically does not matter how you select actions (!) • Off-policy learning: learns optimal q-values, not the values of the policy you are following. • On-policy vs. off-policy: Chapter 5 on RL textbook.

22. Exploration / Exploitation • Several schemes for forcing exploration: • Simplest: random actions ( greedy). • Every time step, flip a coin. • With probability , act randomly. • With probability 1-, act according to current policy. • Regret: expected gap between rewards during learning and rewards from optimal action. • Q-learning with random actions will converge to optimal values, but possibly very slowly, and will get low rewards on the way. • Results will be optimal but regret will be large. • How to make regret small?

23. Exploration Functions • When to explore: • Random actions: explore a fixed amount • Better ideas: explore areas whose badness is not (yet) established, explore less over time • One way: exploration function. • Takes a value estimate and a count, and returns an optimistic utility, (exact form not important):

24. Q-Learning • Q-learning produces tables of q-values:

25. Q-Learning • In realistic situations, we cannot possibly learn about every single state! • Too many states to visit them all in training. • Too many states to hold the q-tables in memory. • Instead, we want to generalize: • Learn about some small number of training states from experience. • Generalize that experience to new, similar states. • This is a fundamental idea in machine learning, and we will see it over and over again!

26. Example: Pacman • Let’s say we discover through experience that this state is bad. • In naïve Q-learning, we know nothing about this state or its q-states. • Or even this one!

27. Feature-Based Representations • Solution: describe a state using a vector of features (properties). • Features map from states to real numbers that capture important properties of the state. • Example features: • Distance to closest ghost/dot. • Number of ghosts. • 1 / (dist to dot)2 • Is Pacman in a tunnel? (0/1) • Is it the exact state on this slide? • Can also describe a q-state (s, a) with features (e.g. action moves closer to food).

28. Linear Feature Functions • Using a feature representation, can write a q-function (or value function) for any state using a few weights: • Advantage: our experience is summed up in a few powerful numbers  • Disadvantage: states may share features but actually be very different in value 

29. Function Approximation • Q-learning with linear q-functions: • Intuitive interpretation: • Adjust weights of active features. • E.g. if something unexpectedly bad happens, do not prefer all states with that state’s features. • Formal justification: online least squares. Exact Q’s Approximate Q’s

30. Example: Q-Pacman

31. 26 24 22 20 30 40 20 30 20 10 10 0 0 Linear Regression 40 20 0 0 20 Prediction Prediction

32. Overfitting 30 25 Degree 15 polynomial 20 15 10 5 0 -5 -10 -15 0 2 4 6 8 10 12 14 16 18 20

33. Policy Search

34. Policy Search • Problem: often the feature-based policies that work well are not the ones that approximate V or Q best. • E.g. value functions may provide horrible estimates of future rewards, but they can still produce good decisions. • Will see distinction between modeling and prediction again later in the course. • Solution: learn the policy that maximizes rewards rather than the value that predicts rewards. • This is the idea behind policy search, such as what controlled the upside-down helicopter.

35. Policy Search • Simplest policy search: • Start with an initial linear value function or q-function. • Nudge each feature weight up and down and see if your policy is better than before. • Problems: • How do we tell the policy got better? • Need to run many sample episodes! • If there are a lot of features, this can be impractical 