Using Inaccurate Models in Reinforcement Learning

1. Using Inaccurate Models in Reinforcement Learning Pieter Abbeel, Morgan Quigley and Andrew Y. Ng Stanford University

2. Overview • Reinforcement learning in high-dimensional continuous state-spaces. • Model-based RL: Difficult to build an accurate model. • Model-free RL: Often requires large numbers of real-life trials. • We present a hybrid algorithm, which requires only • an approximate model, • a small number of real-life trials. • Resulting policy is (locally) near-optimal. • Experiments on flight simulator and real RC car.

3. Reinforcement learning formalism • Markov Decision Process (MDP) • M = (S, A, T, H, s0, R ). • S = n (continuous state space) • Time varying, deterministic dynamics • T = { ft : S x A!S, t = 0,…,H}. • Goal: find policy  : S!A, that maximizes • U() = E [R (st) |  ]. • Focus: task of trajectory following. H t=0

4. Motivating Example • Student-driver learning to make a 90 degree right turn • Only a few trials needed. • No accurate model. • Student-driver has access to: • Real-life trial. • Crude model. • Result: good policy gradient estimate.

5. Algorithm Idea • Input to algorithm: approximate model. • Start by computing the optimal policy according to the model. Real-life trajectory Target trajectory The policy is optimal according to the model, so no improvement is possible based on the model.

6. Algorithm Idea (2) • Update the model such that it becomes exact for the current policy.

7. Algorithm Idea (2) • Update the model such that it becomes exact for the current policy.

8. Algorithm Idea (2) • The updated model perfectly predicts the state sequence obtained under the current policy. • We can use the updated model to find an improved policy.

9. Algorithm • Find the (locally) optimal policy  for the model. • Execute the current policy  and record the state trajectory. • Update the model such that the new model is exact for the current policy . • Use the new model to compute the policy gradient  and update the policy:  :=  + . • Go back to Step 2. Notes: • The step-size parameter  is determined by a line search. • Instead of the policy gradient, any algorithm that provides a local policy improvement direction can be used. In our experiments we used differential dynamic programming.

10. Performance Guarantees: Intuition • Exact policy gradient: • Model based policy gradient: Evaluation of derivatives along wrong trajectory Derivative of approximate transition function Our algorithm eliminates one (of two) sources of error.

11. Performance Guarantees • Let the local policy improvement algorithm be policy gradient. Notes: • These assumptions are insufficient to give the same performance guarantees for model-based RL. • The constant K depends only on the dimensionality of the state, action, and policy (), the horizon H and an upper bound on the 1st and 2nd derivatives of the transition model, the policy and the reward function.

12. Experiments • We use differential dynamic programming (DDP) to find control policies in the model. • Two Systems: • Flight Simulator RC Car

13. Flight Simulator Setup • Flight simulator model has 43 parameters (mass, inertia, drag coefficients, lift coefficients etc.). • We generated “approximate models” by randomly perturbing the parameters. • All 4 standard fixed-wing control actions: throttle, ailerons, elevators and rudder. • Our reward function quadratically penalizes for deviation from the desired trajectory.

14. Flight Simulator Movie

15. Flight Simulator Results 76% utility improvement over model-based approach desired trajectory model-based controller our algorithm

16. RC Car Setup • Control actions: throttle and steering. • Low-speed dynamics model with state variables: • Position, velocity, heading, heading rate. • Model estimated from 30 minutes of data.

17. RC Car: Open-Loop Turn

18. RC Car: Circle

19. RC Car: Figure-8 Maneuver

20. Related Work • Iterative Learning Control: • Uchiyama (1978), Longman et al. (1992), Moore (1993), Horowitz (1993), Bien et al. (1991), Owens et al. (1995), Chen et al. (1997), … • Successful robot control with limited number of trials: • Atkeson and Schaal (1997), Morimoto and Doya (2001). • Robust control theory: • Zhou et al. (1995), Dullerud and Paganini (2000), … • Bagnell et al. (2001), Morimoto and Atkeson (2002), …

21. Conclusion • We presented an algorithm that uses a crude model and a small number of real-life trials to find a policy that works well in real-life. • Our theoretical results show that----assuming a deterministic setting and assuming a reasonable model----our algorithm returns a policy that is (locally) near-optimal. • Our experiments show that our algorithm can significantly improve on purely model-based RL by using only a small number of real-life trials, even when the true system is not deterministic.

23. Motivating Example • Student-driver learning to make a 90 degree right turn • Only a few trials needed. • No accurate model. • Key aspects • Real-life trial: shows whether turn is wide or short. • Crude model: turning steering wheel more to the right results in sharper turn, turning steering wheel more to the left results in wider turn. • Result: good policy gradient estimate.