Reinforcement Learning

1. Reinforcement Learning

2. The study of thinking. 1) Problem-Solving 2) Reasoning

3. Perception Memory Thinking/Cognition Sensation Encoding Retrieval --------------------------------------------------------------------- Low Level Higher Level Thinking is a higher-level cognitive process that requires all sorts of cognitive operations (e.g. attention, perception, memory, language) and is often a conscious, controlled process Should we wait until we understand the lower-level processes first? Research in higher-level cognition might inform research at lower-level cognition and vice-versa.

4. The study of thinking Modern view: • Thinking is an internal cognitive process • The exact nature of these processes cannot be observed directly from behavior • However, most cognitive theories lead to testable predictions. Behavioral experiments can test these predictions. Cognitive processes areinferred indirectly from behavior.

5. Well-defined & Ill-defined Problems Well-defined problems have completely specified initial conditions, goals, and operators  works well with computer simulation Ill-defined problems have some aspects which are not completely specified  sometimes requires insight to see problem in a new way 1. Writing a good paper = ? 2. solving an algebra problem = ? 3. conducting a statistical significance test = ? 4. designing a good experiment = ? 5. choosing a president = ? 6. reducing drunk driving = ? 7. being a nice person = ?

6. - given state • goal state • obstacles • operators Well-defined problem solving INITIAL STATE GOAL STATE INITIAL STATE GOAL STATE ? Play the game: http://www.mazeworks.com/hanoi/

7. problem solving strategies How to solve the maze? - trial and error - forward - backward - means-end analysis

8. Most problem solving situations involves a combination of planning (means-end analysis), trial and error, and reinforcement learning and perhaps ... insight • Reinforcement learning  grew out of behaviorism • Insight  Gestaltists view • Planning  grew out of AI and cognitive psychology

9. R3 S R2 R1 Learning by Reinforcement Associationist theories of thinking -> thinking as response learning • Three elements of associationist theory: • stimulus: a problem solving situation • 2) response: a particular problem solving behavior • 3) associations: strength between stimulus and response

10. Thorndike’s work on cats in a puzzle box • Cats initially solved the puzzle box problem by trial and error – trying various responses until one accidentally worked • After being placed in the box many times, it learned the successful response and pulled the string almost immediately

11. Habit Family Hierarchy Try most dominant response first, then second strongest, etc.

12. Law of exercise: practice tends to increase S-R link • 2) Law of effect: responses that solve a problem increase in strength. Responses that do not help solve problem lose strength R3 S R2 R1

13. What about response chains? • E.g.: • How can path from initial state to goal state be strengthened? How to avoid dead-ends? • How can we reward a successful action that only much later in time leads to success?  problem of delayed reinforcement • Modern reinforcement learning involves passingstrengths of successful responses back through a chain. start goal

14. Maze example • Reinforcement learning example for mazes

15. Reinforcement Learning • Behavior follows simple associations in response chains. No planning, no mental maps, no “insight” • Learning from very simple feedback: failure or success • Associative strengths between response chains are learned. Passing strength back in time start goal

16. Demo’s Reinforcement learning in mazes: http://www.ise.pw.edu.pl/~cichosz/rl-java/ Reinforcement learning in robot-arm control: http://www.fe.dis.titech.ac.jp/~gen/robot/robodemo.html Robot learning task of pole-balancing and devilsticking: http://www-clmc.usc.edu/movies/learning.html

17. Some Amazing Anagrams

18. Stimulus Response (a new letter combination) S R1 g r o w n g o r w n R2 w r o n g R3 w r g n o R4 … Anagram solving time depends on: - familiarity of goal word - letter transition probability of goal word - letter transition probability of presented word - number of moves

20. Class Experiment • Replicate effect of familiarity

21. Ready...? • nrdki • (drink 7.0) • aewtr • (water 3.0) • cahtb • (batch 16.0) • milbc • (climb 7.5) • kcler • (clerk 17.5) • rtypa • (party 14.0) • huocg • (cough 23.5) • rmcap • (cramp 12.0)

22. nrdki • (drink 7.0) • aewtr • (water 3.0) • cahtb • (batch 16.0) • milbc • (climb 7.5) • kcler • (clerk 17.5) • rtypa • (party 14.0) • huocg • (cough 23.5) • rmcap • (cramp 12.0) Mean solution times: High familiarity = 7.9 sec Low familiarity = 17.3 sec

23. Can all thinking be described by trial and error/ stimulus-response? • What about insight?  Gestaltist view • What about planning?  AI view

24. The Handcuffs Puzzle The Set-Up For this puzzle you need two people, some rope and some empty space to do the puzzle in. Each person will need a piece of rope with a loop tied in both ends, so it can be worn as handcuffs. The rope should be reasonably long, so that the person wearing it can easily step over it if they want.    Each person puts on a complete set of handcuffs. Before putting them on, they loop their handcuffs around each other so they are tied together. Each person should wear a complete set of handcuffs. They then have to get themselves apart while following these rules:  The handcuffs cannot be removed.  Do not break, cut, saw through, bite through or in any other way damage the rope. Damaging each other is probably a bad idea too. content copied from: http://ccins.camosun.bc.ca/~jbritton/jbhandcuff.htm