Task Encoding and Strategy Learning in Large Worlds Dagstuhl Workshop, 29 July 2010

1. Task Encoding and Strategy Learning in Large WorldsDagstuhl Workshop, 29 July 2010 Subramanian Ramamoorthy Institute of Perception, Action and Behaviour School of Informatics University of Edinburgh

2. Motivation: Simon’s Ant What does this tell you about robust autonomy in large worlds?

3. A Robust Autonomous Agent • How does she represent the task in order to be able to deploy it in a wide • variety of previously unseen and unmodelled environments? • 2. How is this efficiently utilized for learning?

4. Some Hypotheses (my approach) • Agent represents tasks/environments in terms of a hierarchy of abstractions, ranging from weak sufficient conditions (qualitative information) to detailed quantitative information • Qualitative descriptions define an abstract problem that is useful for coarse reasoning about the large world • Quantitative information can be dealt with locally or at a slower time scale • Variety of learning methods can be combined to leverage their strengths • An ideal abstraction is such that one can make many useful inferences at the abstract level, without recourse to quantitative details that are uncertain/unobserved/undefined • Different from ‘mere’ clustering of states, etc. • We want a decision making strategy to be fully defined at each level

5. So, what does an abstraction look like?Worked Example: Global Control of Cart-Pole

6. Introducing the Cart-Pole System • System consists of two subsystems – pendulum and cart on finite track • Only one actuator – cart • We want global asymptotic stability of 4-dim system • The Game: Experimenter hits the pole with arbitrary velocity at any time, system picks controls • What are the weak sufficient conditions defining this task? Phase space of the pendulum

7. Dealing with the 'Adversary'- Global Structure Adversary could push system anywhere, e.g., here Can describe global strategy as a qualitative transition graph Larger disturbances could truly change quantitative details, e.g., any number of rotations around origin The uncontrolled system converges to this point We want to reach and stay here

8. Describing Local Behaviour: Templates Lemma (Spring – Mass - Positive Damping): Let a system be described by where, and Then it is asymptotically stable at (0,0). Lemma (Spring – Mass - Negative Damping): Let a system be described by where, and Then it has an unstable fixed-point at (0,0), and no limit cycle.

9. The control law: if Balance else if Pump else Spin Constraints: Global Controller for Pendulum

10. The Global Control Law The switching strategy: If then Balance else if then Pump else Spin

11. Demonstration on a physical set-up S. Ramamoorthy, B.J. Kuipers, Qualitative heterogeneous control of higher order systems, Hybrid Systems: Computation and Control (2003)

12. A Few Points to Take Away • No learning in this example but we can still learn things from it • ‘Symbol’  local system with well defined dynamical properties • could also do this using automated formal methods [Shults+Kuipers,AIJ97] • We can talk about task achievement for an entire family of dynamical systems • Weak commitment to functional forms of f and g(also very large parameter intervals, etc.) • Possibility for composition and interactive strategies at symbolic level • Current work: how does one make general relational/logical statements about the behaviour of such models – so that we can use ‘reasoning’ tools at abstract levels • Could enable greedy learning of local models with interesting predicates

13. What does this have to do with learning control strategies?

14. Task: Walking on Irregular Terrain • No detailed models of dynamics • Precisely specified footfalls • Height/length variations • Hard to represent & achieve with state of the art methods!

15. Good Ideas in Two-legged Locomotion

16. Compass Gait Walking: A Conceptual View [Kuo, Science ’05]

17. Natural Parameterization of Gait

18. Abstract Plan Define qualitative strategy in low-dimensions (finite horizon optimal control) (X,U,W) (X,W) (X,U) (X) Lift resulting strategy to the more complex c-space (presently unknown!) S. Ramamoorthy, B.J. Kuipers, Qualitative hybrid control of dynamic bipedal walking, Robotics: Science and Systems II, pp. 89-96 (2006)

19. Trajectory Generation: Multi-link Legged Robot • Random actions • Imperfect gait • Active learning Known Analytically

20. Approximating Unknown Manifold from Data Organize data in a k-NN graph Where is manifoldin the graph? • Manifold  Set of geodesic trajectories restricted to it • If the manifold encodes task – every geodesic must behave like template plan • Diagram must commute! • Minimize commutativity error

21. Result: Controlled Dynamic Walking S. Ramamoorthy, B. Kuipers, Trajectory generation for dynamic bipedal walking through qualitative model based manifold learning, ICRA 08

22. Can We Proceed Without the Low-dim Model? • Consider high-dim data drawn from an unknown low-dim manifold • We can approximate the tangent space: • This can be learnt with a pair of optimization steps • Simple example: 3-link arm • The following error term defines the manifold: • Another error minimization defines geodesic paths:

23. Learnt Skill Manifold: 3-link Arm • The grey mesh is the Delaunay triangulation of the 100 data points • shown for visualization of the desired manifold • (from which curves in fig. c are drawn) I. Havoutis, S. Ramamoorthy, Geodesic trajectory generation on learnt skill manifolds, ICRA 2010

24. Constrained Trajectory Generation on Skill Manifolds

25. Constrained Walking:Variable Foot Placement Following the unconstrained geodesics, oblivious to obstacles Constrained geodesic trajectory – avoid obstacles, while staying within demonstrated class I. Havoutis, S. Ramamoorthy, Constrained geodesic trajectory generation on approximately optimal skill manifolds, IROS 2010

26. What about really dynamic environments?

27. An Adversarial Navigation Problem • Let us make the abstract spaces concrete • you are driving over a network of highways • Two sources of uncertainty: • Oncoming traffic (changing goals) • Changing dynamics, navigability/costs

28. Solution Strategy • In ‘simple’/reasonably well understood worlds, acquire basis strategies • e.g., imitation learning • Could also be more bottom-up exploratory learning • In a continually changing complex world, learn strategies in a game against a (fictitious) adversary

29. One Way to Learning Primitive Strategies • Learn policy from expert: • RL problem • Reward as weighted combination of features • 2-player zero-sum game • select distribution over actions to maximise V(ψ) – V(πE) • nature varies R(s) through weights w [Syed & Schapire 2008]

30. Game-theoretic Strategy Learning • Environment picks transition function, reward • You pick mixture over basis strategies (finite horizon) • Online regret minimization to compute strategies • Composing elemental strategies in response to changing environment

31. Learning to Drive in Novel Scenarios B. Rosman, S. Ramamoorthy, A game theoretic procedure for learning hierarchically structured strategies, ICRA 2010.

32. Learning to Drive in Novel Scenarios

33. Summary • Many autonomous agent behaviours admit efficient descriptions in terms of consistent hierarchy of abstractions • Challenges for learning: • What unsupervised learning methods can we use to extract base concepts (how descriptive are these models)? • Are there principled ways to refine these models over time? • Efficient methods for online strategy learning: how best to define games at the abstract level so they are consistent with fully quantitative local problems? • Life-long and social learning

34. One Future Direction • Complex manipulation problems in terms of hierarchies of abstractions • At one level, one is only thinking of relational concepts: single hole • At another level, one if faced with the full challenge of robotics – grasping, etc. • Ability to work with partial specifications and concepts • Ability to refine representations as we have more and more experience B. Rosman, S. Ramamoorthy, Learning spatial relationships between objects (Under Review)