Reinforcement Learning

1. Reinforcement Learning HUT Spatial Intelligence course August/September 2004 Bram Bakker Computer Science, University of Amsterdam bram@science.uva.nl

2. Overview day 1 (Monday 13-16) • Basic concepts • Formalized model • Value functions • Learning value functions • In-class assignment & discussion

3. Overview day 2 (Tuesday 9-12) • Learning value functions more efficiently • Generalization • Case studies • In-class assignment & discussion

4. Overview day 3 (Thursday 13-16) • Models and planning • Multi-agent reinforcement learning • Other advanced RL issues • Presentation of home assignments & discussion

5. Machine Learning • What is it? • Subfield of Artificial Intelligence • Making computers learn tasks rather than directly program them • Why is it interesting? • Some tasks are very difficult to program, or difficult to optimize, so learning might be better • Relevance for geoinformatics/spatial intelligence: • Geoinformatics deals with many such tasks: transport optimization, water management, etc.

6. Classes of Machine Learning techniques • Supervised learning • Works by instructing the learning system what output to give for each input • Unsupervised learning • Clustering inputs based on similarity (e.g. Kohonen Self-organizing maps) • Reinforcement learning • Works by letting the learning system learn autonomously what is good and bad

7. Some well-known Machine Learning techniques • Neural networks • Work in a way analogous to brains, can be used with supervised, unsupervised, reinforcement learning, genetic algorithms • Genetic algorithms • Work in a way analogous to evolution • Ant Colony Optimization • Works in a way analogous to ant colonies

8. What is Reinforcement Learning? • Learning from interaction • Goal-oriented learning • Learning about, from, and while interacting with an external environment • Learning what to do—how to map situations to actions—so as to maximize a numerical reward signal

9. Some Notable RL Applications • TD-Gammon: Tesauro • world’s best backgammon program • Elevator Control: Crites & Barto • high performance elevator controller • Dynamic Channel Assignment: Singh & Bertsekas, Nie & Haykin • high performance assignment of radio channels to mobile telephone calls • Traffic light control: Wiering et al., Choy et al. • high performance control of traffic lights to optimize traffic flow • Water systems control: Bhattacharya et al. • high performance control of water levels of regional water systems

10. Relationships to other fields Artificial Intelligence Planning methods Control Theory and Operations Research Psychology Reinforcement Learning (RL) Neuroscience Artificial Neural Networks

11. Recommended literature • Sutton & Barto (1998). Reinforcement learning: an introduction. MIT Press. • Kaelbling, Littmann, & Moore (1996). Reinforcement learning: a survey. Artificial Inteligence Research, vol. 4, pp. 237--285.

12. Complete Agent • Temporally situated • Continual learning and planning • Agent affects the environment • Environment is stochastic and uncertain Environment action state reward Agent

13. Supervised Learning Training Info = desired (target) outputs Supervised Learning System Inputs Outputs Error = (target output – actual output)

14. Reinforcement Learning (RL) Training Info = evaluations (“rewards” / “penalties”) RL System Inputs Outputs (“actions”) Objective: get as much reward as possible

15. Key Features of RL • Learner is not told which actions to take • Trial-and-Error search • Possibility of delayed reward • Sacrifice short-term gains for greater long-term gains • The need to explore and exploit • Considers the whole problem of a goal-directed agent interacting with an uncertain environment

16. What is attractive about RL? • Online, “autonomous” learning without a need for preprogrammed behavior or instruction • Learning to satisfy long-term goals • Applicable to many tasks

17. Some RL History Trial-and-Error learning Temporal-difference learning Optimal control, value functions Hamilton (Physics) 1800s Thorndike () 1911 Secondary reinforcement () Shannon Samuel Minsky Bellman/Howard (OR) Holland Klopf Witten Werbos Barto et al. Sutton Watkins

18. Elements of RL • Policy: what to do • Maps states to actions • Reward: what is good • Value: what is good because it predicts reward • Reflects total, long-term reward • Model: what follows what • Maps states and actions to new states and rewards

19. An Extended Example: Tic-Tac-Toe X X X X X O X X X O X X X O X X O X O O O X O O X O O O } x’s move ... x x x } o’s move ... ... ... x o o o x x } x’s move x ... ... ... ... ... } o’s move Assume an imperfect opponent: —he/she sometimes makes mistakes } x’s move x o x x o

20. current state various possible next states * An RL Approach to Tic-Tac-Toe 1. Make a table with one entry per state: State V(s) – estimated probability of winning .5 ? 2. Now play lots of games. To pick our moves, look ahead one step: .5 ? x . . . . . . 1 win x x x o o . . . . . . 0 loss x o o x o . . . . . . o o x 0 draw x x o x o o Just pick the next state with the highest estimated prob. of winning — the largest V(s); a greedy move. But 10% of the time pick a move at random; an exploratory move.

21. RL Learning Rule for Tic-Tac-Toe “Exploratory” move

22. How can we improve this T.T.T. player? • Take advantage of symmetries • representation/generalization • Do we need “random” moves? Why? • Do we always need a full 10%? • Can we learn from “random” moves? • Can we learn offline? • Pre-training from self play? • Using learned models of opponent? • . . .

23. How is Tic-Tac-Toe easy? • Small number of states and actions • Small number of steps until reward • . . .

24. r r r . . . . . . t +1 t +2 s s t +3 s s t +1 t +2 t +3 a a a a t t +1 t +2 t t +3 RL Formalized

25. The Agent Learns a Policy • Reinforcement learning methods specify how the agent changes its policy as a result of experience. • Roughly, the agent’s goal is to get as much reward as it can over the long run.

26. Getting the Degree of Abstraction Right • Time steps need not refer to fixed intervals of real time. • Actions can be low level (e.g., voltages to motors), or high level (e.g., accept a job offer), “mental” (e.g., shift in focus of attention), etc. • States can be low-level “sensations”, or they can be abstract, symbolic, based on memory, or subjective (e.g., the state of being “surprised” or “lost”). • Reward computation is in the agent’s environment because the agent cannot change it arbitrarily.

27. Goals and Rewards • Is a scalar reward signal an adequate notion of a goal?—maybe not, but it is surprisingly flexible. • A goal should specify what we want to achieve, not how we want to achieve it. • A goal must be outside the agent’s direct control—thus outside the agent. • The agent must be able to measure success: • explicitly; • frequently during its lifespan.

28. Returns Episodic tasks: interaction breaks naturally into episodes, e.g., plays of a game, trips through a maze. where T is a final time step at which a terminal state is reached, ending an episode.

29. Returns for Continuing Tasks Continuing tasks: interaction does not have natural episodes. Discounted return:

30. An Example Avoid failure: the pole falling beyond a critical angle or the cart hitting end of track. As an episodic task where episode ends upon failure: As a continuingtask with discounted return: In either case, return is maximized by avoiding failure for as long as possible.

31. Another Example Get to the top of the hill as quickly as possible. Return is maximized by minimizing number of steps reach the top of the hill.

32. A Unified Notation • Think of each episode as ending in an absorbing state that always produces reward of zero: • We can cover all cases by writing

33. The Markov Property • A state should retain all “essential” information, i.e., it should have the Markov Property:

34. Markov Decision Processes • If a reinforcement learning task has the Markov Property, it is a Markov Decision Process (MDP). • If state and action sets are finite, it is a finite MDP. • To define a finite MDP, you need to give: • state and action sets • one-step “dynamics” defined by state transition probabilities: • expected rewards:

35. Value Functions • The value of a state is the expected return starting from that state; depends on the agent’s policy: • The value of taking an action in a stateunder policy p is the expected return starting from that state, taking that action, and thereafter following p :

36. Bellman Equation for a Policy p The basic idea: So: Or, without the expectation operator:

37. Gridworld • Actions: north, south, east, west; deterministic. • If action would take agent off the grid: no move but reward = –1 • Other actions produce reward = 0, except actions that move agent out of special states A and B as shown. State-value function for equiprobable random policy; g = 0.9

38. Optimal Value Functions • For finite MDPs, policies can be partially ordered: • There is always at least one (and possibly many) policies that is better than or equal to all the others. This is an optimal policy. We denote them all by p *. • Optimal policies share the same optimal state-value function: • Optimal policies also share the same optimal action-value function: This is the expected return for taking action a in state s and thereafter following an optimal policy.

39. Bellman Optimality Equation for V* The value of a state under an optimal policy must equal the expected return for the best action from that state: is the unique solution of this system of nonlinear equations.

40. Bellman Optimality Equation for Q* is the unique solution of this system of nonlinear equations.

41. Why Optimal State-Value Functions are Useful Any policy that is greedy with respect to is anoptimal policy. Therefore, given , one-step-ahead search produces the long-term optimal actions. E.g., back to the gridworld:

42. What About Optimal Action-Value Functions? Given , the agent does not even have to do a one-step-ahead search:

43. Solving the Bellman Optimality Equation • Finding an optimal policy by solving the Bellman Optimality Equation exactly requires the following: • accurate knowledge of environment dynamics; • we have enough space and time to do the computation; • the Markov Property. • How much space and time do we need? • polynomial in number of states (via dynamic programming methods), • BUT, number of states is often huge (e.g., backgammon has about 10**20 states). • We usually have to settle for approximations. • Many RL methods can be understood as approximately solving the Bellman Optimality Equation.

44. Temporal Difference (TD) Learning • Basic idea: transform the Bellman Equation into an update rule, using two consecutive timesteps • Policy Evaluation: learn approximation to the value function of the current policy • Policy Improvement: Act greedily with respect to the intermediate, learned value function • Repeating this over and over again leads to approximations of the optimal value function

45. Q-Learning: TD-learning of action values

46. Exploration/Exploitation revisited • Suppose you form estimates • The greedy action at t is • You can’t exploit all the time; you can’t explore all the time • You can never stop exploring; but you should always reduce exploring action value estimates

47. e-Greedy Action Selection • Greedy action selection: • e-Greedy: { . . . the simplest way to try to balance exploration and exploitation

48. Softmax Action Selection • Softmax action selection methods grade action probs. by estimated values. • The most common softmax uses a Gibbs, or Boltzmann, distribution: “computational temperature”

49. Pole balancing learned using RL

50. Improving the basic TD learning scheme • Can we learn more efficiently? • Can we update multiple values at the same timestep? • Can we look ahead further in time, rather than just use the value at the next timestep? • Yes! All these can be done simultaneously with one extension: eligibility traces