AMAM Conference 2005

1. AMAM Conference 2005 Adaptive Motion in Animals and Machines

2. Outline of the talk • Short AMAM conference overview • Introduction to Embodied Artificial Intelligence (keynotes, R. Pfeifer) • More detailed look at: • Sensory Motor Coordination • Value-Systems

3. AMAM: Conference Overview • Motivation of studying Biology • Source of inspiration for robotics • Model features of rather simple animals (insects…) • Robots and animals have to solve the same physical problems • Robots are useful tools for computational neuroscience • Testing Neural Models within a complete sensing-acting loop

4. Biorobotics • Bio-inspired technologies • New sensors: Whiskers and Antennas • Muscle-Like (flexible) actuators • Flexible robotic arms and hands • Biped and humanoid robots • Numerical Models of animal and human locomotion • Central Pattern Generator based and other control methods • Some robots for illustratoin:

5. AMAM: robots • Scorpion [Kirchner05] • 8 legged robot • BigDog [Buehler, Boston Dynamics]

6. AMAM: Robots • Fish Robot • Iida • Stumpy • „Special“ robot to investigate cheap design locomotion (Iida)

7. AMAM Conference: Robots • ZAR 4 [boblan05] • Bionic robot arm driven by artificial muscels • And many more: • Insects : • Coackroaches[ritzmann05] • Worm [menciassi05] • Amoebic Robots [ishiguro05] • Bisam Rat [albiez05]

8. Embodied Artificial Intelligence[Pfeifer99, Iida03] • Not interested in the control aspects of robots alone, but rather in designing entire systems • Morphology, Materials + Control • Synthetic Methology: Understanding intelligent behavior by building • Concentrate on complete autonomous robots • Self-Sufficient: Sustain itself over a extended period of time • Situatedness: acquires all information about the environment from its own sensory system • „Lives“ in a specified ecological niche: no need for universal robots • Embodiment: real physical agents • Adaptivity • „Why do plants have no brain? They do not move.“ [Brooks] • Often aspects of only simple animals are modeled by robots (locomotion of insects…) • It took evolution 3 billion years to evolve insects/legged locomotion, but only 500 million more years to develop humans • => locomotion must be a hard problem

9. Embodied AI: Principles • Emergence: • Emergent Behaviours: „emerge“ by the interaction of the robot with the environment • Not preprogrammed • Agent is the result of its history • Exploit the dynamics of the system • More adaptive : developmental mechanisms • Diversity Compliance: • Exploiting ecologicol niche / behavioral diversity • Exploration/Exploitation trade off

10. Embodied AI: Principles • Parallel, loosely coupled processes • Intelligence emerge from a lager number of parallel processes • Processes are connected to the agent‘s sensory-motor aparatus • Coupling through embodiment or coordination • No functional decompositon/hierarchical control like in traditional robotic • Supsumption architecture [brooks86] • Sensory-Motor Coordination • Structuring sensory input • Generation of good sensory-motor patterns: • Correlated • Stationarity • Can simplify learning • Dimensionality Reduction of sensory-motor space [lungeralla05, boekhorst03]

11. Embodied AI: Principles • „Morphological“ Computation • Parts of the control can be „computed“ by the morphology • Facets in flies, motion paralax • Springs and flexible material • Exploit system dynamics for control • E.g. Exploit gravity and flexible actuators • Can simplify control considerably • Increase learning speed by morphology • „Extreme“ Example: Passive dynamic walker • Cheap Design: • Exploit physics and constraints of ecological niche • Use the most simple architecture for a given task

12. Embodied AI: Principles • Redundancy: • Overlap of functionality in the subsystems • Sensory system, Motor system • Required for diversity and adaptivity • Ecological Balance: • Complexity of the sensory, motor and neural system has to match for a given task • Balance between morphology, materials and control [Ishiguro03] • Value Principle • Motivation of the robot to do something (should be more general than RL) • Essential for every complete autonomous agent • No generally accepted solution exists • 2 approaches will be discussed in more detail

13. Traditional Robotics / AI • In difference to traditional robotics • Limited numbers of degrees of freedom (e.g. wheels) • Stiff structure and joints (servo motors) • Easy to control • All Computation has to be done by the control system • Limited natural dynamics • Centralized rule-based control • Functional decomposition • „Sense-think-act“ cycle • Problems: • Frame problem • Symbol grounding problem

14. Sensory-Motor coordination (SMC)[Pfeifer99, Lungarella05] • Used for categorization • Traditional approach: Sensory-input to category mapping • Prototype or example matching • Difficulties: Often this mapping is not learnable • Noise and Inaccuracies in Sensors • Ambigious sensory input (Type 2 problems)

15. Categorization: Example [Nolfi97] • Learn 2 categories (Wall, Cylinder) with IR sensors • Data for: • 180 orientations, 20 distances • Learn with neural network • Just linear output units • 4 resp 8 hidden neurons • Very bad results: 35 % correct categorization Back dots: correct categoritization

16. SMC: Categorization • Approach the problem through interacting with the environment • Object related actions to structure the input • Simplifies the problem of categorization • No real internal category representation • Just different behaviors for different categories • Empirical studies about Dimensionality Reduction [lungarella05] • Example in infants: Look at object from different directions in the same distance

17. SMC: Example • Learning optimal categorization strategy through a genetic algorithm • Nolfi‘s experiment: • Fitness: Time the robot is near the cylinder • Evolved Behavior: • Robot never stops in front of target: • Move back/forth and left/right hand side

18. SMC: Example • Learning to distinguish circles and diamonds [Beer96] • Catching circles, avoiding diamonds • Agent can only move horizontally • Again evolved controller

19. SMC: Example • Results: • Not merely centering and statically pattern matching • Dynamic strategy, with active scanning • Both policies evolve sensory-motor coordination strategies • Examples show quite good the idea of sensory-motor coordination • Other examples: • Darwin II [Reeke89] • Garbage Collector [Pfeifer97, Schleier96] Catching Circle Avoiding Diamond

20. SMC: Conclusion • Nice new ideas for categorization tasks and robotics in generell • Simple examples that illustrate the use of SMC for categorization • Examples are „well-suited“ for SMC • No complex categorization problem (e.g for visual object recognition) found in the literature • Only numerical results which proofs dimensionality reduction • How to use them? • Critic: Humans are also able to do categorization very well without sensory-motor interaction • The emphasis of SMC is a bit overstressed by the authors

21. Value Systems & Developmental Learning [oudeyer04/05, steels03] • Intrinsic Motivation of the Agent: • learn more about the environment • Ideal case: open-end learning • Many different behaviors may emerge • Very adaptive • 2 approaches to this problem discussed in more detail • Intelligent Adaptive Curiosity (IAC) [oudeyer04] • Autotelic Principle [steels03] • Still in the beginning, only for toy examples • Other approaches comming from RL • Intrinsically motivated RL [singh04] • Self Motivated Development [schmidhuber05]

22. IAC: Motivation • Push agent towards situations in which it maximizes learning progress • Balance between the „unknown“ and the „predictable“ • Goal: Improve prediction machine • A(t) … action • SM(t)… sensory-motor context • S(t+1)… prediction

23. IAC: framework • Prediction error • => Decrease E(t) • First naive approach • Learning Progress • Em(t)… mean Error at time t • Do not reward high error values, reward high LP • Meta Learning Machine (predicts error) • Choose action which maximizes Learning Progress • Problem ?

24. IAC: • Problem of naive approach: • Transition from complex, not predictable situations to simple situations is considered as learning progress • Solution: • Instead of comparing the LP succesive in time, compare the LP succesive in state space

25. IAC: algorithm • Prediction machine P • Consists of a set of local experts. • Each expert consists of training examples • Simple NN algorithm is used for prediction • Build kd-tree incrementally : experts in the leaves • Each expert stores prediction errors and the mean • Calculate local learning progress • LPi(t) = -(Empi(t) – Empi(t – DELAY) • Used for action selection • Very simple algorithms used • More sophisticated algorithms have a good chance to improve performance

26. IAC: experiments • Toy example: • 2 wheeled robot, can produce sound • Toy: position depends on sound frequency intervall • f1 : moves randomly • f2 : stops moving • f3 : toy jumps to robot • Predictor: predict relative position of the toy

27. IAC: experiments • Results: • Basically 3 experts • First explores intervall f3, then intervall f2 • f1 is not explored : not predictable

28. IAC: experiments • Playground experiment • AIBO robot on a baby play mat • Various toys: can be bitten, bashed or simply detected

29. IAC: Playground Experiment • Motor Control: • Turning head (2 DoF, pan + tilt) • Bashing (2 DoF, strength + angle) • Crouch + Bite (1 DoF, crouches given distance in direction it is looking at) • Perception: • 3 High level sensors (just binary values) • Visual object detection • Biting Sensor • Infra-red distance sensor • Bashing + Biting only produce visible results if applied in front of an appropriate object • Agent knows nothing about sensorimotor affordances

30. IAC: Results • Different stages evolves • Stage 1: random exploration + body babbling • Stage 2: Most of the time looking around (no biting + bashing) • Stage 3: biting and bashing • Sometimes produces something, robot still not oriented to objects • Stage 4: Starts to look at objects • Learns precise location of the object • Stage 5: Trying bite biteable object, trying to bash bashable object

31. The Autotelic principle [steels03] • Autotelic activities: no real reward • Climbing, painting… • Motivational driving signal comes from the individual itself • Balance between high challenge and required skill • too high: withdrawal • too low: boredom • Operational description given in [steels03], no real experiments found

32. Autotelic Principle: Operational Descripion • Agent: • Organised in number of sub-agencies (components) • Establish input/output mapping based on knowledge • Each component must be parameterized to self adjust challenge levels • Precision of movement, weights of objects… • Parameter vector pi for each component • Goal: not to reach a stable state, keep exploring parameter landscape • Each component has also an associated skill vector

33. Autotelic Principle: Operational Descripion • Self Regulation: • Operation phase: Clamp challenge parameters, learn skills through learning • Shake-Up phase: • Increase challenge: skill level already too high • Decrease challenge: performance could not be reached

34. Conclusion: Value Systems • Both approaches try to create open-ended learner • Interesting ideas • Only very simple algorithms used, or not even implemented • Open for improvement • Can help to structure learning progress in complex environments • Complete autonomous agents will need some sort of developmental value system • No complex real-world experiments found • Scalable?

35. Conclusion: Embodied Intelligence • Provides new ways of thinking about robotic / intelligence in general • Provides a better understanding of intelligent behavior by modelling the behavior. • Good principles to design an agent • Claims to solve many problems of traditionial AI • Good and promising ideas • Somehow the algorithmic solutions for more complex systems are missing • Actually: same problems as for traditional AI • Works for small problems • Hard to scale up

36. The End • Thank you!

37. Literature • [pfeifer99] R. Pfeifer and C. Schleier, Understanding Intelligence, MIT Press • [iida03] F. Iida and R. Pfeifer, Embodied Artificial Intelligence • [kirchner05] D. Spenneberg, F. Kirchner, Embodied Categorization of spatial environments on the Basis of Proprioceptive Data, AMAM 2005 • [ritzmann05] R. Ritzmann, R. Quinn, Convergent Evolution and locomotion through complex terrain by insects, vertebrates and robots, AMAM 2005 • [menciassi05] A. Menciassi, S. Spina, Bioinspired robotic worms for locomotion in unstructered environments, AMAM2005 • [ishiguro05] A. Ishiguro, M. Shimizu, Slimebot: A Modular robot that exhibits amoebic locomotion, AMAM2005 • [albiez05] J. Albiez, T. Hinkel, Reactive Foot-control for quadruped walking, AMAM2005 • [boblan05] I. Boblan, R. Bannasch, A Humanlike Robot Arm and Hand with fluidic muscles: The human muscle and the control of technical realization, AMAM 2005 • [lungeralla05] M. Lungarella, O. Sporns, Information Self-Structuring: Key Principle for Learning and Development • [broekhorst03] R. Broekhorst, M. Lungarella, Dimensionality Reduction through sensory motor-coordination

38. Literature • [ishiguro03] A. Ishiguro, T. Kawakatsu, How should control and body systems be coupled? A robotic case study, Embodied artificial intellingence 2003 • [nolfi97] S. Nolfi, Evolving non-trivial behavior on autonomous robots: Adaptation is more powerful than decompositionand integration • [beer96] R. Beer, Toward the Evolution of Dynamical Neural Networks for Minimally Cognitive Behavior • [reeke89] G. Reeke, O. Sporns, Synthetic neural modeling: A multilevel approach to analysis of brain complexity • [pfeifer97] R. Pfeifer, C. Schleier, Sensory-motor coordination: The metaphor and beyond: Practice and future of autonmous robots • [schleier96] C. Schleier, D. Lambrinos, Categorization in a real world agent using haptic exploration and active perception • [oudeyer04] P. Oudeyer, F. Kaplan, Intelligent Adaptive Curiosity: a source of Self-Development • [oudeyer05] P. Oudeyer, F. Kaplan, The Playground Experiment: Task independent development of a curious robot. • [steels03] L. Steels, The Autotelic Principle • [singh04] S. Singh, A. Barto, Intrinsically Motivated Learning of Hierarical Collections of Skills • [schmidhuber05] J. Schmidhuber, Self-Motivated Development Through Rewards for Predictor Errors/Improvements

39. Measure influence of SMC [lungeralla05, broekhorst03] • New experiments with SMC • Measure the effect of SMC with information processing quantities • Experiments of Broekhorst: • Robot: • Wheeled • CCD camera (compressed to 10 x 10 pixels) • IR sensors (12) • Measure angular velocity • 5 different Experiments: • Control setup: Move forward • Moving object • Wiggling : Move forward in oscillatory movement • Tracking 1: Move forward + track object • Tracking 2: Move forward + track moving object • Preprogrammed control

40. Measure Influence of SMC [broekhorst03] • Quantify dimension of the sensory information • Measure Correlation on most significant principal components from the different modalities (R*) • 3 different information quantities • Shannon entropy • Dominance of the highest eigenvector • Number of PC‘s that explain 95% of variance …Eigenvalue of R*

41. Results: • Difference: • Variance in the experiments • SMC experiments have higher variance • SMC experiments and non SMC experiments can be distinguished • No further straithforward results

42. Measure Influence of SMC [lungarella05] • Experimental Setup: • Active Vision: (compressed 55 x 75 pixels) looking at screen • 2 behaviors: • Foveation: „follow red area“ • Random: Same motion structure, not coordinated • 2 scenarios • Artificial Scene: Random Data with moving red block • Natural Images

43. Measure Influence of SMC [lungarella05] • Quantify sensory information • Entropy • Joint-Entropy • Mutual Information • Integration : Multivariate Mutual Information • Complexity : • Quantify Dimensionality Reduction • PCA • Isomap ([tenenbaum01], also recognizes non-linear dimensions)

44. Results for foveation behavior • Entropy in central regions decreased • Mutual information increased

45. Results for foveation behavior • Integration and Complexity where much larger in the center

46. Results for foveation behavior • Reduced dimensionality (isomap) • Mutual information between center and motor actions also increased