Georgios’s Visions ( interactive learning  representations )

1. HHMM… Georgios’s Visions(interactive learning  representations) MIT CSAIL

2. Ed Wood(Characterized as the worst film maker ever) "Home? I have no home Hunted,despised, Living like an animal! The jungle is my home. But I will show the world that I can be its master! I will perfect my own race of people. A race of atomic supermen which will conquer the world!"

3. Why? • Learning from delayed reward is hopeless (in my opinion) • Supervised learning is impractical • Humans and animals live in societies • Need something above RL and below supervised learning

4. Social learning Interactive learning Learning to communicate Classroom learning Competitive learning Do what I mean not what I say What do you mean? Let’s talk Robot apprentices Searching for the right representations Possible Titles

5. Final Product PHYSICAL ENVIRONMENT Observations, Actions,Rewards, State modification Erik’s representation Georgios’s representation Pavlov’s representation

6. Obstacles • A mathematical framework for interactive learning (reward shaping?) • What are objects (sensory, motor sequences ?) • How do they relate to each other. What are the representations (atomic, propositional, first-order?)

7. Example Systems • A robot that learns to navigate by interaction with a human trainer • A personalized web agent(active information extraction) • Personal assistants (office)

8. Tools & Concepts • H-POMDPS? • What is missing? • Dynamic abstractions (structure learning) • Teleological abstractions • Relational structure • Factorization (hierarchical reuse) • Multiagency /concurrency

9. Grounded Projects • Other H-POMDP applications • Model reduction in POMDPs with macros • Structure learning of H-POMDPs • Theoretical localization results in grid-worlds with structure • Mathematical framework for interactive learning • Efficient algorithms for learning stochastic models

10. Other H-POMDP Applications • Passive “hierarchical” HMM applications • Policy recognition (AMM) (Hung Bui) • Video Structure discovery (HHMM) (Lexing Xie) • Human activity recognition (Nuria Oliver) • Emotion Recognition (multi –level HMM) (Ira Cohen) • Natural English text & cursive hand-writing (HHMM) (Fine) • Information extraction (HHMM) (skounakis) • Active recognition/learning • Active object detection/recognition (RL) (Lucas paletta) • Selective perception policies for guiding sensing (layered HMM ) (Nuria Oliver, Eric Horvitz) • Active learning of HMMs (Tobias Scheffer) • What can we do (active learning?) (active recognition==POMDP planning?) • Recognition of office activity / Active recognition of office activity / Active learning of model parameters

11. POMDPs & Macro-Actions • A model based RL over a dynamic grid abstraction in belief space with macro-actions (NIPS 2003) • Consider only needed part of belief space • Learn faster than just using primitive actions • Ability to do information gathering • What’s next? • A new minimized POMDP other than than the belief state representation (PSRs? Non-linear dimensionality reductions? Smaller HMMs?) • Other domains

12. Structure Learning • Natural Language approaches • Sequitor (Nevill-Manning) • Unsupervised Language acquisition (Carl G. de Marcken) • Structure learning in graphical models • Discovering hidden state (X. Boyen) • From Data Mining • Bursty and Hierarchical structure in streams (Jon Kleinberg)

13. Localizing in Flat Grid Worlds is NP-hard • In flat POMDPs finding localization plans that are within a log factor of optimal is NP-Hard (Sven Koenig) • Does the same hold for H-POMDPs?

14. Mathematical Framework for Interactive learning ENVIRONMENT State s T State s Reward r R Reward r O Policy z AGENT Supervisor Action a

15. Interactive Learning Literature • Programmable RL agents (David Andre) • Principle methods for advising RL agents (Garrison Cottrell) • Machine discovery of effective admissible heuristics (Armand E. Prieditis) • Supervised learning combined with an actor-critic architecture (Michaels Rosenstein) • Shaping in RL by changing the physics of the problem (Jette Randolv) • What if the teacher needs to learn too?

16. Efficient Learning Algorithms for Models of Stochastic Processes • Parameter learning in graphical models is inefficient (structure learning impractical) • Can we do better? • Train model where it needs to be trained • Do informed searching when learning structure

17. Conclusions • Big results require big ambitions • To make progress towards AI,We need to make learning and planning more interactive • This will keep me busy for a while