Concurrent Hierarchical Reinforcement Learning

1. Concurrent Hierarchical Reinforcement Learning

2. Example problem: Stratagus • Challenges • Large state space • Large action space • Complex policies

3. Writing agents for Stratagus Game company : N person-years to write “AI” for game Hierarchical RL : concurrent partial program + learning optimal completion Standard RL – learn from tabula rasa

4. Outline • Running example • Background • Our contributions • Concurrent Alisp language for partial programs • Basic learning method • … with threadwise decomposition • … and temporal decomposition • Results

5. Running example • Peasants can move, pickup and dropoff • Penalty for collision • Cost-of-living each step • Reward for dropping off resources • Goal : gather 10 gold + 10 wood

6. Outline • Running example • Background • Our contributions • Concurrent Alisp language for partial programs • Basic learning method • … with threadwise decomposition • … and temporal decomposition • Results

7. Reinforcement Learning a Learning algorithm s,r Policy

8. Q-functions • Represent policies implicitly using a Q-function • Qπ(s,a) = “Expected total reward if I do action a in environment state s and follow policy π thereafter” Fragment of example Q-function

9. Hierarchical RL and ALisp Partial program a Learning algorithm s,r Completion

10. Program state includes Program counter Call stack Global variables Single-threaded Alisp program (defun top () (loop do (choose ‘top-choice (gather-wood) (gather-gold)))) (defun gather-wood () (with-choice ‘forest-choice (dest *forest-list*) (nav dest) (action ‘get-wood) (nav *base-loc*) (action ‘dropoff))) (defun gather-gold () (with-choice ‘mine-choice (dest *goldmine-list*) (nav dest)) (action ‘get-gold) (nav *base-loc*)) (action ‘dropoff))) (defun nav (dest) (until (= (pos (get-state)) dest) (with-choice ‘nav-choice (move ‘(N S E W NOOP)) (action move))))

11. Q-functions • Represent completions using Q-function • Joint state ω = env state + program state • MDP + partial program = SMDP over {ω} • Qπ(ω,u) = “Expected total reward if I make choice u in ω and follow completion π thereafter” Example Q-function

12. Temporal Q-decomposition Top Top Top GatherGold GatherWood Nav(Mine2) Nav(Forest1) Qr Qc Qe Q • Temporal decompositionQ = Qr+Qc+Qewhere • Qr(ω,u) = reward while doing u • Qc(ω,u) = reward in current subroutine after doing u • Qe(ω,u) = reward after current subroutine

13. Q-learning for ALisp • Temporal decomposition => state abstraction • E.g., while navigating, Qr doesn’t depend on gold reserves • State abstraction => fewer parameters, faster learning, better generalization • Algorithm for learning temporally decomposed Q-function (Andre+Russell 02) • Yields optimal completion of partial program

14. Outline • Running example • Background • Our contributions • Concurrent Alisp language for partial programs • Basic learning method • … with threadwise decomposition • … and temporal decomposition • Results

15. Multieffector domains • Domains where the action is a vector • Each component of action vector corresponds to an effector • Stratagus: each unit/building is an effector • #actions exponential in #effectors

16. HRL in multieffector domains • Single Alisp program • Partial program must essentially implement multiple control stacks • Independencies between tasks not used • Temporal decomposition is lost • Separate Alisp program for each effector • Hard to achieve coordination among effectors • Our approach - single multithreaded partial program to control all effectors

17. Threads and effectors Get-gold Get-wood • Threads = tasks • Each effector assigned to a thread • Threads can be created/destroyed • Effectors can be moved between threads • Set of effectors can change over time Defend-Base Get-wood

18. Concurrent Alisp syntax • Extension of ALisp • New/modified operations : • (action (e1 a1) ... (en an)) • each effector eidoes ai • (spawnthread fn effectors) • start new thread that calls function fn, and reassign effectors to it • (reassigneffectors dest-thread) • reassign effectors to thread dest-thread • (my-effectors) • return list of effectors assigned to calling thread • Program should also specify how to assign new effectors on each step to a thread

19. Concurrent Alisp syntax • Extension of ALisp • New/modified operations : • (action (e1 a1) ... (en an)) • each effector eidoes ai • (spawnthread fn effectors) • start new thread that calls function fn, and reassign effectors to it • (reassigneffectors dest-thread) • reassign effectors to thread dest-thread • (my-effectors) • return list of effectors assigned to calling thread • Program should also specify how to assign new effectors on each step to a thread

20. Concurrent Alisp syntax • Extension of ALisp • New/modified operations : • (action (e1 a1) ... (en an)) • each effector eidoes ai • (spawnthread fn effectors) • start new thread that calls function fn, and reassign effectors to it • (reassigneffectors dest-thread) • reassign effectors to thread dest-thread • (my-effectors) • return list of effectors assigned to calling thread • Program should also specify how to assign new effectors on each step to a thread

21. Concurrent Alisp syntax • Extension of ALisp • New/modified operations : • (action (e1 a1) ... (en an)) • each effector eidoes ai • (spawnthread fn effectors) • start new thread that calls function fn, and reassign effectors to it • (reassigneffectors dest-thread) • reassign effectors to thread dest-thread • (my-effectors) • return list of effectors assigned to calling thread • Program should also specify how to assign new effectors on each step to a thread

22. Concurrent Alisp syntax • Extension of ALisp • New/modified operations : • (action (e1 a1) ... (en an)) • each effector eidoes ai • (spawnthread fn effectors) • start new thread that calls function fn, and reassign effectors to it • (reassigneffectors dest-thread) • reassign effectors to thread dest-thread • (my-effectors) • return list of effectors assigned to calling thread • Program should also specify how to assign new effectors on each step to a thread

23. An example single-threaded ALisp program An example Concurrent ALisp program (defun gather-gold () (with-choice ‘mine-choice (dest *goldmine-list*) (nav dest) (action ‘get-gold) (nav *base-loc*) (action ‘dropoff))) (defun nav (dest) (until (= (my-pos) dest) (with-choice ‘nav-choice (move ‘(N S E W NOOP)) (action move)))) (defun top () (loop do (until (my-effectors) (choose ‘dummy)) (setf peas (first (my-effectors)) (choose ‘top-choice (spawn gather-wood peas) (spawn gather-gold peas)))) (defun gather-wood () (with-choice ‘forest-choice (dest *forest-list*) (nav dest) (action ‘get-wood) (nav *base-loc*) (action ‘dropoff))) (defun top () (loop do (choose ‘top-choice (gather-gold) (gather-wood)))) (defun gather-wood () (with-choice ‘forest-choice (dest *forest-list*) (nav dest) (action ‘get-wood) (nav *base-loc*) (action ‘dropoff)))

24. Concurrent Alisp semantics Peasant 1 (defun nav (Wood1) (loop until (at-pos Wood1) (setf d (choose ‘(N S E W R))) do (action d) )) • defun gather-wood () • (setf dest (choose *forests*)) • (nav dest) • (action ‘get-wood) • (nav *home-base-loc*) • (action ‘put-wood)) Thread 1 Running Paused Waiting for joint action Making joint choice (defun nav (Gold2) (loop until (at-pos Gold2) (setf d (choose ‘(N S E W R))) do (action d) )) (defun gather-gold () (setf dest (choose *goldmines*)) (nav dest) (action ‘get-gold) (nav *home-base-loc*) (action ‘put-gold)) Peasant 2 Paused Thread 2 Running Making joint choice Waiting for joint action Environment timestep 27 Environment timestep 26

25. Concurrent Alisp semantics • Threads execute independently until they hit a choice or action • Wait until all threads are at a choice or action • If all effectors have been assigned an action, do that joint action in environment • Otherwise, make joint choice • Can be converted to a semi-Markov decision process assuming partial program is deadlock-free and has no race conditions

26. Q-functions • To complete partial program, at each choice state ω, need to specify choices for all choosing threads • So Q(ω,u) as before, except u is a joint choice Example Q-function

27. Outline • Running example • Background • Our contributions • Concurrent Alisp language for partial programs • Basic learning method • … with threadwise decomposition • … and temporal decomposition • Results

28. Making joint choices at runtime • Large state space • Large choice space • At state ω, want to choose argmaxu Q(ω,u) • # joint choices exponential in # choosing threads • Use relational linear value function approximation with coordination graphs (Guestrin et al 2003) to represent, maximize Q efficiently

29. Basic learning procedure • Apply Q-learning on equivalent SMDP • Boltzmann exploration, maximization on each step done efficiently using coordination graph • Theorem: in tabular case, converges to the optimal completion of the partial program

30. Outline • Running example • Background • Our contributions • Concurrent Alisp language for partial programs • Basic learning method • … with threadwise decomposition • … and temporal decomposition • Results

31. Problems with basic learning method • Temporal decomposition of Q-function lost • No credit assignment among threads • Suppose peasant 1 drops off some gold at base, while peasant 2 wanders aimlessly • Peasant 2 thinks he’s done very well!! • Significantly slows learning as number of peasants increases

32. Threadwise decomposition • Idea : decompose reward among threads (Russell+Zimdars, 2003) • E.g., rewards for thread j only when peasant j drops off resources or collides with other peasants • Qj(ω,u) = “Expected total reward received by thread j if we make joint choice u and make globally optimal choices thereafter” • Threadwise Q-decomposition Q = Q1+…Qn

33. Learning threadwise decomposition Peasant 3 thread Peasant 2 thread Peasant 1 thread r2 r1 r3 Top thread r decomp a Action state

34. Learning threadwise decomposition Peasant 3 thread Q-update Peasant 2 thread Q-update Peasant 1 thread Q2(ω,·) Q-update Q1(ω,·) Q3(ω,·) Top thread + argmax = u Q(ω,·) Choice state ω

35. Outline • Running example • Background • Our contributions • Concurrent Alisp language for partial programs • Basic learning method • … with threadwise decomposition • … and temporal decomposition • Results

36. Threadwise and temporal decomposition Peasant 3 thread Top thread Peasant 2 thread • Qj = Qj,r + Qj,c + Qj,e where • Qj,r (ω,u) = Expected reward gained by thread j while doing u • Qj,c (ω,u) = Expected reward gained by thread j after u but before leaving current subroutine • Qj,e (ω,u) = Expected reward gained by thread j after current subroutine Peasant 1 thread

37. Outline • Running example • Background • Our contributions • Concurrent Alisp language for partial programs • Basic learning method • … with threadwise decomposition • … and temporal decomposition • Results

38. Resource gathering with 15 peasants Reward of learnt policy Flat Num steps learning (x 1000)

39. Resource gathering with 15 peasants Undecomposed Reward of learnt policy Flat Num steps learning (x 1000)

40. Resource gathering with 15 peasants Threadwise Undecomposed Reward of learnt policy Flat Num steps learning (x 1000)

41. Resource gathering with 15 peasants Threadwise + Temporal Threadwise Undecomposed Reward of learnt policy Flat Num steps learning (x 1000)

42. Conclusion • Contributions • Concurrent ALisp language for partial programs • Algorithms for choice selection, learning • Threadwise and temporal decomposition of Q-function • Encode prior knowledge and use structure in Q-function for fast learning in very complex multieffector domains Questions?