1 / 30

Topic 18: Reinforcement Learning

Topic 18: Reinforcement Learning. Coverage in 2014 :This Slideshow : Topic18b’.pptx , Sutton Video: http://videolectures.net/icml09_sutton_itdrl/ may be: http ://videolectures.net/mlss06au_singh_rl / (Introduction to RL)

sarila
Download Presentation

Topic 18: Reinforcement Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Topic 18: Reinforcement Learning Coverage in 2014:This Slideshow: Topic18b’.pptx, Sutton Video: http://videolectures.net/icml09_sutton_itdrl/ may be: http://videolectures.net/mlss06au_singh_rl/ (Introduction to RL) Q&A: Kaebling Paper May be video: Helicopter Control—talk at ICML Introduction Bellman Update Temporal Difference Learning Discussion of Project1 Active Reinforcement Learning Applications Summary Road Map Reinforcement Learning: http://aitopics.org/topic/reinforcement-learning

  2. Introduction Supervised Learning: Example Class Reinforcement Learning: … Situation Reward Situation Reward

  3. Examples Playing chess: Reward comes at end of game Ping-pong: Reward on each point scored Animals: Hunger and pain - negative reward food intake – positive reward

  4. Framework: Agent in State Space Remark: no terminal states Example: XYZ-World e e 1 2 3 R=+5 s s n w 6 R=-9 4 5 R=+3 sw ne s s n x/0.7 x/0.3 8 R=+4 9 R=-6 7 nw Problem: What actions should an agent choose to maximize its rewards? s 10

  5. XYZ-World: Discussion Problem 12 TD P Bellman e e (3.3, 0.5) 1 2 3 R=+5 s s n I tried hard but: any better explanations? w 6 R=-9 4 5 R=+3 sw ne s s n (3.2, -0.5) x/0.7 x/0.3 8 R=+4 9 R=-6 7 nw s (0.6, -0.2) 10 • Explanation of discrepancies TD for P/Bellman: • Most significant discrepancies in states 3 and 8; minor in state 10 • P chooses worst successor of 8; should apply operator x instead • P should apply w in state 6, but only does it only in 2/3 of the cases; which affects the utility of state 3 • The low utility value of state 8 in TD seems to lower the utility value of state 10  only a minor discrepancy P: 1-2-3-6-5-8-6-9-10-8-6-5-7-4-1-2-5-7-4-1.

  6. XYZ-World: Discussion Problem 12 Bellman Update g=0.2 e e 10.145 20.72 30.58R=+5 s s n w 6-8.27R=-9 40.03 53.63R=+3 sw ne s s n x/0.7 x/0.3 83.17R=+4 9-5.98R=-6 70.001 nw s 100.63 • Discussion on using Bellman Update for Problem 12: • No convergence for g=1.0; utility values seem to run away! • State 3 has utility 0.58 although it gives a reward of +5 due to the immediate penalty that follows; we were able to detect that. • Did anybody run the algorithm for other g e.g. 0.4 or 0.6 values; if yes, did it converge to the same values? • Speed of convergence seems to depend on the value of g.

  7. XYZ-World: Discussion Problem 12 TD inverse R TD e e (0.57, -0.65) 1 2 3 R=+5 s s n (2.98, -2.99) w 6 R=-9 4 5 R=+3 sw ne s s n x/0.7 x/0.3 8 R=+4 (-0.50, 0.47) 9 R=-6 7 nw s • Other observations: • The Bellman update did not converge for g=1 • The Bellman update converged very fast for g=0.2 • Did anybody try other values for g (e.g. 0.6)? • The Bellman update suggest a utility value for 3.6 for state 5; what does this tell us about the optimal policy? E.g. is 1-2-5-7-4-1 optimal? • TD reversed utility values quite neatly when reward were inversed; x become –x+u with u[-0.08,0.08]. (-0.18, -0.12) 10 P: 1-2-3-6-5-8-6-9-10-8-6-5-7-4-1-2-5-7-4-1.

  8. XYZ-World --- Other Considerations • R(s) might be known in advance or has to be learnt. • R(s) might be probabilistic or not • R(s) might change over time --- agent has to adapt. • Results of actions might be known in advance or have to be learnt; results of actions can be fixed, or may change over time. • One extreme: everything is known Bellman Update; other extreme: nothing is known except states are observable, and available actions are known TD-learning/Q-learning

  9. Basic Notations • T(s,a,s’) denotes the probability of reaching s’ when using action a in state s; it describes the transition model • A policy p specifies what action to take for every possible state sS • R(s) denotes the reward an agent receives in state s • Utility-based agents learn an utility function of states uses it to select actions to maximize the expected outcome utility. • Q-learning, on the other hand, learns the expected utility of taking a particular action a in a particular state s (Q-value of the pair (s,a) • Finally, reflex agents learn a policy that maps directly from states to actions • RL-Glossary: http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node9.html Stanford Blackboard Video on this topic: http://freevideolectures.com/Course/2257/Machine-Learning/16

  10. Reinforcement Learning “You use your brain or a computer” Introduction Bellman Update Temporal Difference Learning Project 1 Active Reinforcement Learning Applications Summary “You learn about the world by performing actions in it”

  11. 2. Bellman Equation Utility values obey the following equations: U(s) = R(s) + γ*maxaΣs’T(s,a,s’)*U(s’) Assume γ =1, for this lecture! “measure utility in the future, after apply action a” Can be solved using dynamic programming. Assumes knowledge of transition model T and reward R; the result is policy independent! Video on “foundations” of RL: http://videolectures.net/mlss06au_singh_rl/

  12. Bellman Update If we apply the Bellman update indefinitely often, we obtain the utility values that are the solution for the Bellman equation!! Ui+1(s) = R(s) + γ maxa(Σs’(T(s,a,s’)*Ui(s’))) Some Equations for the XYZ World: Ui+1(1) = 0+ γ*Ui(2) Ui+1(5) = 3+ γ *max(Ui(7),Ui(8)) Ui+1(8) = 4+ γ *max(Ui(6),0.3*Ui(7) + 0.7*Ui(9) ) Bellman Update:

  13. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Project1 Active Reinforcement Learning Applications Summary

  14. 3. Temporal Difference Learning Idea: Use observed transitions to adjust values in observed states so that the comply with the constraint equation, using the following update rule: UΠ (s)  UΠ (s) + α [ R(s) + γ UΠ (s’) - UΠ (s) ] α is the learning rate; γ discount rate Temporal difference equation. No model assumption --- T and R have not to be known in advance.

  15. Updating Estimations Based on Observations: New_Estimation = Old_Estimation*(1-) + Observed_Value* New_Estimation= Old_Estimation + Observed_Difference* Example: Measure the utility of a state s with current value being 2 and observed values are 3 and 3 and the learning rate is: =0.2: Initial Utility Value:2 Utility Value after observing 3: 2x0.8 + 3x0.2=2.2 Utility Value after observing 3,3: 2.2x0.8 +3x0.2= 2.36 Remark: This is the simplest form of TD-Learning

  16. Temporal Difference Learning Idea: Use observed transitions to adjust values in observed states so that the comply with the constraint equation, using the following update rule: UΠ (s)  (1-α)*UΠ (s) + α*[ R(s) + γ UΠ (s’)] α is the learning rate; γ discount rate Temporal difference equation. No model assumption --- T and R have not to be known in advance.

  17. History and Summary: http://en.wikipedia.org/wiki/Q-learning TD-Q-Learning Goal: Measure the utility of using action a in state s, denoted by Q(a,s); the following update formula is used every time an agent reaches state s’ from s using actions a: Q(a,s)  Q(a,s) + α [ R(s) + γ*maxa’Q(a’,s’) - Q(a,s) ] • α is the learning rate; g is the discount factor • Variation of TD-Learning • Not necessary to know transition model T&R!

  18. Example: Simplified PD World e 1 2 P w s n w 4 3 D e Ungraded Homework State Space: {1, 2, 3, 4} Policy: agent starts is state 1 and applies operators: e-p-w-e-s-d-w-e-w-n-e-p-s-d Construct the q-table assuming =0.4 and =0.2! Solution will be discussed in the Jan. 27 lecture!

  19. TD-Learning Itself After each step, this is what I have obtained: Q(e,1) = -0.4 Q(p,2) = 4.8 Q(w,2) = -0.432 Q(e,1) = -0.64 Q(s,2) = -0.4 Q(d,3) = 4.8 Q(w,3) = -0.4 Q(e,4) = -0.432 Q(w,3) = -0.64 Q(n,4) = -0.4512 Q(e,1) = -0.4=-0.64*0.6+ 0.4*(-1+0.2x4.8) Q(p,2) = 7.648 • Q(s,2) = -0.256= =-0.4*0.6+ 0.4*(-1+0.2x4.8) Q(d,3) = 7.6288 • Next loop 4.8 will • be 7.6, making q(e,1) • and q(s,2) much larger!

  20. TD-Learning Itself • In general, TD Learning is a prediction technique • In Project3 in 2013, we used it to predict state utilities • where an agent obtains feedback about a world by • exploring the states of the world by applying actions • http://www.scholarpedia.org/article/Temporal_difference_learning

  21. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Project1 Active Reinforcement Learning Applications Summary

  22. 5. Active Reinforcement Learning Now we must decide what actions to take. Optimal policy: Choose action with highest utility value. Is that the right thing to do?

  23. Active Reinforcement Learning No! Sometimes we may get stuck in suboptimal solutions. Exploration vs Exploitation Tradeoff Why is this important? The learned model is not the same as the true environment.

  24. Explore vs Exploit Exploitation: Maximize its reward vs Exploration: Maximize long-term well being.

  25. Simple Solution to the Exploitation/Exploration Problem • Choose a random action once in k times • Otherwise, choose the action with the highest expected utility (k-1 out of k times)

  26. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Project1 Active Reinforcement Learning Applications Summary

  27. 6. Applications http://www.youtube.com/watch?v=ICgL1OWsn58 http://www.robots-dreams.com/2010/10/22nd-kondocup-robot-soccer-khr-class-video.html Robot Soccer Game Playing Checker playing program by Arthur Samuel (IBM) http://en.wikipedia.org/wiki/Arthur_Samuel Update rules: change weights by difference between current states and backed-up value generating full look-ahead tree Backgrammon(5:37): http://www.youtube.com/watch?v=RvgEA1XJiY8

  28. Applications2 Elevator Control Helicopter Control Demo: http://heli.stanford.edu/ ICML 2008 Talk: http://videolectures.net/icml08_coates_lcmd

  29. 7. Summary • Goal is to learn utility values of states and • an optimal mapping from states to actions. • If the world is completely known and does • not change, we can determine utilities by • solving Bellman Equations. • Otherwise, temporal difference learning has • to be used that updates values to • match those of successor states. • Active reinforcement learning learns the • optimal mapping from states to actions.

  30. Sutton ICML 2009 Video on 4 Key Ideas of RL Link:http://videolectures.net/icml09_sutton_itdrl/ We will show the video 10:34-45:37 • Discussion: What is unique about RL? • Our Discussion found: • Rewards instead of giving the correct answer • Importance of Exploration and Sampling • Adaptation: Dealing with Changing Words • Other Things not mentioned on 4/15/2014: worried about the future/lifelong well-being; finding good policies,…

More Related