Reinforcement Learning for the Soccer Dribbling Task

1. Reinforcement Learning for the Soccer Dribbling Task Arthur Carvalho Renato Oliveira

2. Introduction • RoboCup soccer simulation • Scoring • “A Data Mining Approach to Solve the Goal Scoring Problem” • Passing • “A New Passing Strategy Based on Q-Learning Algorithm in RoboCup” • Dribbling • ?

3. Soccer Dribbling Task

4. Outline • The soccer dribbling task as a RL problem • RL solution • Experiments • Conclusion

5. The Soccer Dribbling Task as a RL Problem • Coach • Setting positions • Dribbler is placed in the center-left region together with the ball • Adversary is placed in a random position • Manage the play • Adversary wins when he gains possession or when the ball goes out of the field • Dribbler wins when he crosses the field with the ball

6. The Soccer Dribbling Task as a RL Problem • When an episode ends, the coach starts a new one • RoboCup soccer simulator operates in discrete time steps • Episodic reinforcement-learning framework

7. The Soccer Dribbling Task as a RL Problem • Actions • HoldBall() • Dribble(α, k) • Dribble(30, 5), Dribble(330, 5), Dribble(0, 5), Dribble(0, 10) • The dribbler can kick the ball forward (strongly and weakly), diagonally upward, and diagonally downward.

8. The Soccer Dribbling Task as a RL Problem

9. Outline • The soccer dribbling task as a RL problem • RL solution • Experiments • Conclusion

10. RL Solution • Sarsa + CMAC • Sarsa estimates the action-value function for the current policy and for all state-action pairs

11. RL Solution • CMAC • Partitioning the state space into several receptive fields (hyper-rectangles) • Each one is associated with a weight • Multiple partitions of the state space (layers) are usually used • The CMAC’s response to a given input is equal to the sum of the weights of the excited receptive fields

12. RL Solution

13. Outline • The soccer dribbling task as a RL problem • RL solution • Experiments • Conclusion

14. Experiments • RoboCup soccer simulator • Version 14.0.3, protocol 9.3 • 20m x 20m region • -greedy policy • = 0.01 • Parameters • Learning rate: • Discount rate:

15. Experiments • Adversary • Fixed policy • It computes a near-optimal interception point (UvATrilearn 2003 team) • Two phases • Training • Testing

16. Experiments • Training Phase: 5 independent runs, each one lasting 50,000 episodes 53%

17. Experiments • Qualitatively • Rule #1

18. Experiments • Qualitatively • Rule #2

19. Experiments • Testing: • 10,000 episodes • Weights with the highest success rate • The weights were not updated • , i.e., the dribbler always selected the action with the highest estimated value • The dribbler won 5,795 episodes (58%)

20. Outline • The soccer dribbling task as a RL problem • RL solution • Experiments • Conclusion

21. Conclusion • Dribble • Soccer dribbling task • Reinforcement learning solution • Benchmark • Start point for dribbling tasks in other sports games • E.g., hockey, basketball, and football

22. Thank you! Source code available at: http://sites.google.com/site/soccerdribbling Arthur Carvalho Renato Oliveira a3carval@cs.uwaterloo.ca rmo@cin.ufpe.br