Controlling Mobile Robots with Distributed Neuro-Biological Systems

1. Controlling Mobile Robots with Distributed Neuro-Biological Systems Sebastian Gutierrez-Nolasco (UCI) Nalini Venkatasubramanian (UCI) Alfredo Weitzenfeld (ITAM) Contact info seguti@ics.uci.edu

2. Biologically Inspired Robotic Systems • Nature has always been a source of inspiration in the development of autonomous robotic systems • Ethology • Animal behavior-based simulation • Interaction with the environment is usually oversimplified • Lack of strong biological basis for their working assumptions • Lack of any formal underpinnings for the simulation results • Neuroethology • behavior related to neurobiological structure • Replicate brain models to provide credible and general animal behavior • Provide inspiration for further robotics architectures • More complex and accurate than ethology systems • Enable experimentation • Experimentation requires real-time performance

3. Neuroethological robotic systems • Super Robots • Incorporate extensive processing capabilities • Bulky • Expensive • Inexpensive Robots • Smaller and inexpensive robots connected to a network of processing nodes • Concerns • Real-time performance • Unpredictable communication environment affects robot performance

4. Challenges of Biologically Inspired Autonomous Robots • A neural model may take hours of processing time • Simulation of multiple neural networks require a distributed processing environment • A typical retina model may consist of more than 100,000 neurons and 500,000 interconnections • Biologically inspired robotics demand sophisticated image processing techniques • Communication intensive tasks are required • Autonomous robotic agents have real-time and processing restrictions, as well as power awareness requirements • Battery usage is a major concern in mobile robots

5. Developing Biologically Inspired Robot Architectures • Developing software for autonomous mobile robots is complex • Highly heterogeneous methods for capturing and processing sensor information • Multiple sensory input devices • Sensory input multi-granularity • Communication is error-prone due to unpredictable interference and failures • partial and complete failures • Unreliability and disconnection • Varying available bandwidth

6. Our Approach • Develop an embedded architecture capable of conducting neuroethological robotic experimentation • Inexpensive small robots communicate (wireless) with distributed computational resources • Neural models are distributed in multiple processing nodes • Adaptive robotic middleware optimizes robot communication in response to varying network conditions

7. Structure of the Talk • Neuroethological Modeling • Study animal behavior and corresponding neural structure as inspiration to robotic architectures • Embedded Mobile Robots • Develop distributed wireless robot architectures capable of efficient neural processing • Adaptive Middleware • Achieve real-time computation and adapt embedded architecture to varying network conditions • Internet Based Robotics • Enable remote robot task development and experimentation

8. Neuroethological Modeling • Study animal behavior and corresponding neural structure as inspiration to robotic architectures • Embedded Mobile Robots • Develop distributed wireless robot architectures capable of efficient neural processing • Adaptive Middleware • Achieve real-time computation and adapt embedded architecture to varying network conditions • Internet Based Robotics • Enable remote robot task development and experimentation

9. Behavior:Frog and Toad - Rana Computatrix [Arbib 1987, Cervantes 1990] - PS MO PS NMO + + + - - PS Mate PS Prey PS Pred Mate-Pair S Prey-Acq S Find-Loc S Pred-Av S Moving-Object S Non-Moving-Object S Perceptual Schema (PS) Main Schema (S)

10. Behavior:Toad Prey Acquisition [Cervantes 1985] Stimulus Response Mobile visual stimulus in lateral visual field (monocular perception) Orientation Mobile visual stimulus in binocular visual field (short distance) Binocular fixation Attack Mechanic stimulus in mouth and pharynx receptors Snap Clean

11. Behavior:Toad Prey Acquisition with Detour Behavior Before and After Learning [Corbacho and Arbib 1995] 10cm Barrier 20cm Barrier After learning 20cm Barrier Before learning

12. Schema Computational Model dout din 1 1 ... ... ... ... Schema dout din m n data in Schema Level 1 data out Schema Level 2 Schema Level Neural Level Other Processes Neural

13. Neural based Behavior:Toad Prey Acquisitions and Predator Avoidance Prey Recognizer Prey Approach Moving Stimulus Selector Predator Recognizer Forward Depth Predator Avoid Visual Orient Backward Sidestep Static Object Recognizer Static Object Avoidance Tactile Schema Level Tectum Neural Level T5_2 R1-R2 R1-R2 R1-R2 R3 R3 Motor Heading Map R3 R4 R4 R4 TH10 Retina Stereo MaxSelector PreTectum/Thalamus

14. 16 15 14 13 10 7 4 12 9 6 3 8 5 11 2 1 Toad Prey Acquisition with Detour:Simulation Results 10cm barrier 20cm barrier Before learning 20cm barrier After learning

15. Neuroethological Modeling • Study animal behavior and corresponding neural structure as inspiration to robotic architectures • Embedded Mobile Robots • Develop distributed wireless robot architectures capable of efficient neural processing • Adaptive Middleware • Achieve real-time computation and adapt embedded architecture to varying network conditions • Internet Based Robotics • Enable remote robot task development and experimentation

16. Embedded Mobile Robots:Robot Hardware LEGO OOPIC

17. Internet Embedded Mobile Robots:Distributed Embedded Architecture Wireless Remote Computaional System Instance 1 Autonomous Robot 1 Internet Server ... ... ... Autonomous Robot N Remote Computational System Instance N

18. Embedded Mobile Robots:Distributed Embedded Architecture • Time consuming processes are carried out in the (neural) computational system • Neural processing • Image processing • Limited task are carried out in the robot hardware • Sensory input • Motor output • Default behavior • Communication and data transformation is managed by the adaptive middleware

19. Embedded Mobile Robots:Distributed Embedded Architecture camera Remote Computaional System servo Frame Grabber Robot Wireless CPU (OOPic) Transceiver Sensors (tact) Transceiver PC Power stage motor motor

20. Embedded Mobile Robots:Distributed Embedded Architecture Remote Computational System Wireless NSL NSL NSL NSL Robot Video/ Image Video Server Processing camera ASL ASL transceiver transceiver tactile Tactile Server ASL ASL motor Motor Server NSL NSL NSL NSL NSL – Neural Simulation Language ASL – Abstract Schema Language

21. Embedded Mobile Robots:Processing cycle Video capture Video processing Model simulation Model output (d , r , c) Navigation control

22. Neuroethological Modeling • Study animal behavior and corresponding neural structure as inspiration to robotic architectures • Embedded Mobile Robots • Develop distributed wireless robot architectures capable of efficient neural processing • Adaptive Middleware • Achieve real-time computation and adapt embedded architecture to varying network conditions • Internet Based Robotics • Enable remote robot task development and experimentation

23. Distributed Systems Middleware • Enables the modular interconnection of distributed software • Abstract over low level mechanisms used to implement resource management services • Concurrent Object Oriented Model • Separation of concerns and reuse of services • Customizable, Composable Middleware Frameworks • Provide for dynamic network and system customizations, dynamic invocation/revocation/installation of services • Concurrent execution of multiple resource management policies

24. Core Resource Management Services • Core Services - basic services where interactions between the application and system can occur. • Building blocks for other services • Reduce interactions among many services to interactions between a few simple services • Choosing core services - commonly observed patterns • Recreation of data/services at a remote site • Capturing approximation of distributed state at multiple sites • Interactions with a global repository

25. System (Meta) Level Replication Access Control DGC Migration Check- pointing Remote Creation Distributed Snapshot Directory Services Application (Base) Level TLAM: The Two Level Meta-architecture

26. Adaptive Robotic Middleware (ARM) • Extends the TLAM to • Optimize information flow between robots and the computational system • Determine how, when and what information should be modified in order to match fluctuations in the communication environment • Compose communication protocols to obtain the combined benefits - conflicting requirements • Explicit knowledge of how communication protocols compose and interact is required • Adapt protocols and mechanisms to changing communication and power constraints

27. ARM: Distributed Embedded Architecture Remote Computational System Wireless ARM Robot NSL NSL NSL NSL Video/ Image Video Server ARM processing camera ASL ASL transceiver transceiver tactile Tactile Server ASL ASL motor Motor Server NSL NSL NSL NSL NSL – Neural Simulation Language ASL – Abstract Schema Language

28. ARM: Components • Communication manager • Provide and enforce application level requirements • Components • Oracle • determine most suitable protocol implementation in terms of coverage and efficiency • Set of communication protocols • Protocol installer/uninstaller • Resident ARM module running in the robot (resident evil) • Adaptation manager • Provide adaptation and monitor mechanisms operating at different levels of abstraction • Reactive • Triggered when failure to achieve intended communication goal is detected • Proactive • Triggered when a more efficient communication can be achieved under the current environment conditions • Adaptation Repository • Determine most suitable adaptation strategy to be applied

29. ARM: Example

30. Neuroethological Modeling • Study animal behavior and corresponding neural structure as inspiration to robotic architectures • Embedded Mobile Robots • Develop distributed wireless robot architectures capable of efficient neural processing • Adaptive Middleware • Achieve real-time computation and adapt embedded architecture to varying network conditions • Internet Based Robotics • Enable remote robot task development and experimentation

31. Interned based Robotics: Web Access

32. Experimental Results: 2 Preys

33. Experimental Results: 2 Preys and Predator

34. (B) (A) (D) (C) Embedded Mobile Robots:Experimental Results: Prey Acquisition with 10 cm Barrier

35. Embedded Mobile Robots:Experimental Results: Prey Acquisition with 20 cm Barrier (B) (D) (C) (A) (F) (H) (G) (E)

36. Neural based Behavior:Prey Acquisition (10cm barrier) • Barrier (PreTectum) • Prey (Tectum) • Integrated (MHM) • Heading (MHM) • Tactile Visual Fields

37. Neural based Behavior:Prey Acquisition (20cm barrier before bumping) • Barrier (PreTectum) • Prey (Tectum) • Integrated (MHM) • Heading (MHM) • Tactile Visual Fields

38. Neural based Behavior:Prey Acquisition (20cm barrier after bumping) • Barrier (PreTectum) • Prey (Tectum) • Integrated (MHM) • Heading (MHM) • Tactile Visual Fields

39. Neural based Behavior:Prey Acquisition (20cm barrier after learning) • Barrier (PreTectum) • Prey (Tectum) • Integrated (MHM) • Heading (MHM) • Tactile Visual Fields

40. Future Work • Complete Internet based System • Develop middleware adaptation capabilities • Build smaller robotic systems • Extend to multiple robot tasks • Extend vision system to “true” moving forms • Extend biological models

41. Video

42. Bonus Section

43. Research Cycle Formal Models Data, Hypotheses Brain Theory Neuroscience Robotics (Experiments) (Modeling) New Ideas New Hypotheses (Results from Experiments with Physical Devices) New Hypothesis Gaps in Knowledge

44. Neural Maps T - Temporal, D - Dorsal, N - Nasal, V - Ventral O - Optic Tectum, B - Nucleus of Belonci C - Lateral Geniculate Nucleus, P - Thalamic Pretectal Neuropil X - Basal Optic Root [Scalia and Fite 1974]

45. s mp mf input neuron output Neuron Model • mp - membrane potential : dmp(t)/dt = f(s,mp,t) • mf - firing rate : mf(t) =s(mp(t)) • Leaky Integrator : t dm(t)/dt = -m(t) + s

46. Retina-Thalamus-Tectum + + Retina TP R4 Input R3 R2 + + + TP – ThalamusPreTectum GL – Glomerelus SN – Stellate Neurons SP – Small Pear LP – Large Pear PY - Pyramidal + GL SN - + + - - SP + + + - LP Synapsis + + + + + Excitation + - Inhibition - PY Output

47. Max Selector [Didday 1976]

48. Max Selector Model MaxSelectorModel MaxSelector out MaxSelector s_in uf Stimulus Vlayer Ulayer in MaxSelector s _out in s _ Output vf in uf u _ in v _ • nslModel MaxSelectorModel () extends NslModel() • { • private MaxSelector maxselector(10); • private MaxSelectorStimulus stimulus(10); • private MaxSelectorOutput output(); • public void initSys() { • system.setRunTime(10.0); • system.setRunDelta(0.1); • } • public void makeConn() { • nslConnect(stimulus.s_out,maxselector.s_in); • nslConnect(stimulus.s_out,output.s_in); • nslConnect(maxselector.out, output.uf); • } • }

49. Max Selector Module MaxSelector out Vlayer Ulayer in s _in vf uf u _in v _in • nslModule MaxSelector (int size) extends NslModule() • { • public Ulayer u1(size); • public Vlayer v1(size); • public NslDinDouble1 in(size); • public NslDoutDouble1 out(size); • public void makeConn(){ • nslRelabel(in,u1.s_in); • nslConnect(v1.vf,u1.v_in); • nslConnect(u1.uf,v1.u_in); • nslRelabel(u1.uf,out); • } • }

50. Vlayer Ulayer s _in vf u _in uf v _in Ulayer and Vlayer Modules • nslModule Vlayer(int size) extends NslModule () • { • public NslDinDouble1 u_in(size); • public NslDoutDouble0 vf(); • private NslDouble0 vp(); • private NslDouble0 hv(); • private double tau; • public void initRun() { • vp =0; • vf = 0; • hv=0.5; • tau=1.0; • } • public void simRun() { • vp = nslDiff(vp,tau,-vp+nslSum(u_in) – hv); • vf = nslRamp(vp); • } • } • nslModule Ulayer(int size) extends NslModule () • { • public NslDinDouble1 s_in(size); • public NslDinDouble0 v_in(); • public NslDoutDouble1 uf(size); • private NslDouble1 up(size); • private NslDouble0 hu(); • private double tau; • public void simRun() { • up =0; • uf = 0; • hu = 0.1; • tau =1.0; • } • public void simRun() { • up =nslDiff(up,tau, -up + uf - v_in – hu + s_in); • uf = nslStep(up,0.1,0.1.0); • } • }