From Neuroscience to Mechatronics

1. Presentation by Fabian Diewald JASS April 2006 From Neuroscience to Mechatronics

2. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

3. first vertebrates ~500 million years ago civilization of man since ~10 000 years Learning from nature – a justified strategy Consequently nature has a great time advantage!

4. The human brain • cerebellum = "little brain" • responsible for accurate movement • instructions by the forebrain insufficient • instructions have to be translated into accurate commands by the cerebellum • cerebellum essential part in learning motor skills

5. Neurons – elementary components of the central nerve system • dendrites: "input" of a neuron • axon: "output" • axon terminals/boutons contact other neurons or muscles

6. Neurons – elementary components of the central nerve system • transmission works electrically • information through the axon is encoded in changes of electrical potential • short impulses with fixed intensity • information is contained in firing frequency("pulse rate modulation")

7. Synapses – link between neurons • gap against electrical transmission • transmission between neurons is controlled by chemicals called neurotransmitters • controlled transmission is important in regard to adaptation and learning

8. climbing fibre: axon of inferior olive; one single climbing fibre for one Purkinje cell wraps around dendritic tree of Purkinje cell; essential for "learning" inferior olive The structure of the cerebellar cortex parallel fibres=axons of granule cells,synapse on Purkinje cells Purkinje cells: ~200 000 synapses per cell, only output for cerebellar cortex granule cells, ~20 per mossy fibre, half of all neurons are granule cells mossy fibres = information input, synapse on granule cells

9. The influential theory of Marr-Albus • twice independently proposed: • Marr, David: “A Theory of Cerebellar Cortex”, 1969, Journal of Physiology 202: 437-470 • Albus, James S.: "A theory of cerebellar function", 1971, Mathematical Biosciences 10: 25-61 • major aspects: • climbing fibre carries a teaching signal • signal influences the synapses of Purkinje cells • consequently the intensity of certain inputs to the Purkinje cells can be controlled

10. The influential theory of Marr-Albus – disappointment by Marr himself • Marr 1982:"In my own case, the cerebellar study (...) disappointed me, because even if the theory was correct, it did not enlighten one about the motor system – it did not, for example, tell one how to go about programming a mechanical arm."

11. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

12. The vestibulo-ocular reflex (VOR) • aim: stable picture of our environment/object we are staring at in spite of moving head • only use of visual information would be too slow for stabilization (visual processing delay about 50-100ms) • signals of the vestibular organ (motion of the head) are more or less directly transmitted to the extra-ocular eye muscles, leading to process times of 5-10ms • vestibular signals are also influenced by visual information

13. The vestibulo-ocular reflex (VOR) – experimentation with monkeys • VOR does an excellent job in monkeys as well as in other vertebrates under normal conditions • magnifying or miniaturizing glasses cause abnormal image motion speed perceived during head motion • monkeys with glasses show problems with head motion in the beginning • after several days: considerable improvement is completed • then: taking the glasses away • monkeys show problems again • after some days: normal behaviour again • unconscious use of VOR, motoric issue, calibration needed • typical case concerning the cerebellum

14. The vestibulo-ocular reflex – paths of information eye (visual information) vestibular organ (head motion) adaptive path over Purkinje cells: transferred on parallel fibres direct path: very quick not accurate (not sensible to changes in the system, e.g. changes in eye muscle strength) inferior olive adaptation path: teaching signal transferred on climbing fibres "strengthens" or "weakens" the synapses synapses ofPurkinje cells Purkinje cells cerebellum eye muscles

15. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

16. camera systems in general face-recognitionsystems The vestibulo-ocular reflex – possible technical applications

17. Affordabledriver-assistance systems The vestibulo-ocular reflex – also an important automotive application • wide angle cameras used to obtain an overview of the environment • for a closer look (road signs, number plates etc.) a telephoto lens is needed, which is quite sensitive to motion

18. Requirements concerning driver-assistance systems • camera stabilization only by optical means is too slow • inertial measurement of head angular velocity needed • affordable hardware • cheap inaccurate sensors must be allowed (multi-sensor fusion of translation/angular velocity and visual information) • inaccuracies have to be compensated automatically •  similar problems/requirements as in biology • the vestibulo-ocular reflex may solve our technical problem!

19. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

20. The VOR – biological and technical analogons How can we implement cerebellar structures as computer hardware? the structure of the cerebellum is quite homogeneous – although different information appears in different regions, these regions are similar idea of "cerebellar chip" today: normally no need for cerebellar chip as DSPs are very multifunctional and cerebellar algorithms can be implemented software-based (e.g. in Matlab)

21. The VOR – biological and technical analogons • vestibular organ in the inner ear 6-DOF inertial measurement unit • direct pathway vestibular to muscles CAN bus system • cerebellum DSP with specific algorithm • extra-ocular eye muscles servo motors mechatronics: neuroscience/biology: main difficulties: commercially available inertial measurement units are too big for driver-assistance systems the algorithm biology uses in the cerebellum has to be detected and implemented, which is obviously difficult(remember Marr's quotation!)

22. Camera stabilization system at TU Munich, Institute of Applied Mechanics • serial gimballed configuration • two perpendicular axes • pan actuator driven by:4.5 Watt motor • inner frame driven by:11 Watt Maxon motor withbacklash free HarmonicDrivegearbox • each axis controlled by differential encoders with 512 steps • camera: CMOS sensor with effective resolution of 750x400 pixels • in the future: only a mirror is moved leading to further miniaturization

23. Coping with difficulty 1:size of inertial measurement unit (IMU) • smallest available intelligent IMU 50x38x25 mm³ • too big for driver-assistance-systems • new IMU was developed within the FORBIAS project: edge length: 15 mm 3 gyroscopes(rotation), bandwith up to 100Hz accelerometer with 3 axes on one chip(translation), bandwith up to 640Hz

24. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

25. Coping with difficulty 2 (main difficulty):implementing an adequate algorithm (1) central ideas/principles: • motor movements are calculated according to angular velocity of measurement unit ("direct path") • theoretically (perfect sensors/perfect production) this is sufficient for camera stabilization • but: due to inaccuracies in production and sensors themselves there is always relative movement of the picture of the environment ("optical flow") • the relation between angular rates and this relative movement is calculated • the result dynamically influences a matrix w

26. Coping with difficulty 2 (main difficulty):implementing an adequate algorithm (2) central ideas/principles: • this matrix w calculates a certain output out of angular velocity • this output is used to "clean" the signal of the direct path (velocity) additively • result: after some time the matrix w is "perfect" (learning completed) and always knows what to add to the measured angular velocity so that there is no optical flow any more, i.e. optical flow and angular velocity are decorrelated • consequently further cost-reduction possible by teaching the system after assembling and storing the matrix w on a chip • but in this case inaccuracies because of e.g. plastic mechanical deformation during use cannot be compensated

27. The vestibulo-ocular reflex – paths of information eye (visual information) vestibular organ (head motion) adaptive path over Purkinje cells: transferred on parallel fibres direct path: very quick not accurate (not sensible to changes in the system, e.g. changes in eye muscle strength) inferior olive adaptation path: teaching signal transferred on climbing fibres "strengthens" or "weakens" the synapses synapses ofPurkinje cells Purkinje cells cerebellum eye muscles

28. Decomposition: splitting physical input signal in adequate inputs for decorrelator (decorrelator should be able to learn by derivatives of velocity as well, not only by velocity!) biological equivalent: parallel fibres Coping with difficulty 2 (main difficulty):implementing an adequate algorithm central idea: decorrelation of angular rates and optical flow, decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow, angular velocity and its derivative (optical flow = teaching signal) Decorrelator with weight matrix ,equivalent to synapses of Purkinje cells Ddecomposition to "parallel fibre signals" Sdiagonal sensitivity matrix brainstem B = direct path I P C w + + - from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive Applications AMAA, Berlin, April 25-27, 2006 (in press)

29. output given by weighted sum: • proportional output p and • derivative path d (added after integrator I) • are calculated out of the "input" angular velocity (vector pf) by the dynamic matrix w • J: Jacobean projecting angular rates on degrees of freedom of motion device Coping with difficulty 2 (main difficulty):implementing an adequate algorithm central idea: decorrelation of angular rates and optical flow, decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow, angular velocity and its derivative (optical flow = teaching signal) Decorrelator with weight matrix ,equivalent to synapses of Purkinje cells Ddecomposition to "parallel fibre signals" Sdiagonal sensitivity matrix brainstem B = direct path I P C w + + - from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive Applications AMAA, Berlin, April 25-27, 2006 (in press)

30. learning rule: • similar to LMS algorithm of adaptive control theory • here: teaching signal = optical flow f; in biology: teaching signal = climbing fibre signals • covariance learning rule (Sejnowski 1977): • If parallel-fibre firing pf (angular velocity and derivative) is positively correlated with climbing-fibre firing f (optical flow), reduce the weight w, • if parallel-fibre firing pf (angular velocity and derivative) is negatively correlated with climbing-fibre firing f (optical flow), increase the weight w, • if parallel-fibre firing pf (angular velocity and derivative) is uncorrelated with climbing-fibre firing f (optical flow), no change Coping with difficulty 2 (main difficulty):implementing an adequate algorithm central idea: decorrelation of angular rates and optical flow, decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow, angular velocity and its derivative (optical flow = teaching signal) Decorrelator with weight matrix ,equivalent to synapses of Purkinje cells Ddecomposition to "parallel fibre signals" Sdiagonal sensitivity matrix brainstem B = direct path I P C w + + - from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive Applications AMAA, Berlin, April 25-27, 2006 (in press)

31. delay ~100ms delay ~10ms for frequencies above ~2.5Hz the phase difference can get quite big => instability in the learning process possible => delay of pf-signal (~100ms) and smoothing filter removing high frequencies ("eligibility trace") Possible variants of the basic systems (1) central idea: decorrelation of angular rates and optical flow, decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow, angular velocity and its derivative (optical flow = teaching signal) Decorrelator with weight matrix ,equivalent to synapses of Purkinje cells Ddecomposition to "parallel fibre signals" Sdiagonal sensitivity matrix brainstem B = direct path I P C w + + - from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive Applications AMAA, Berlin, April 25-27, 2006 (in press)

32. capacity of climbing fibre pathway (teaching signal) is limited => unimportant data flow can be avoided by only evaluating the sign of the teaching signal, i.e. the direction of the optical flow => learning was not significantly worse compared to learning with exact values Possible variants of the basic systems (2) central idea: decorrelation of angular rates and optical flow, decorrelator: contains dynamic state matrix w, dynamically influenced by optical flow, angular velocity and its derivative (optical flow = teaching signal) Decorrelator with weight matrix ,equivalent to synapses of Purkinje cells Ddecomposition to "parallel fibre signals" Sdiagonal sensitivity matrix brainstem B = direct path I P C w + + - from: Günthner, W.; Glasauer, S.; Wagner, P.; Ulbrich, H.: Biologically inspired multi-sensor fusion for adaptive camera stabilization in driver-assistance systems, Advanced Microsystems for Automotive Applications AMAA, Berlin, April 25-27, 2006 (in press)

33. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

34. Other visual technical problems to be solved for driver assistance • optical flow caused by camera rotation can successfully be removed by camera stabilization as just seen, but... • line of sight may have to be kept on the road • rapid changes of viewing direction ("saccades") have to be implemented • furthermore: closer examination of certain objects (signs etc.) need visual tracking

35. Testing of the camera stabilization system under laboratory conditions • testing in the laboratory by mounting the system on a hexapod • created sensed angular rates of ~100°/s • optical flow was reduced from 6 pix/frame to less than 1 pix/frame after 2-3 minutes of adaptation • the improvement within this time was also subjectively viewable

36. Testing of the camera stabilization system "on the road" (1) • system mounted near the rear-view mirror • additional camera installed as well to detect points of interest for the camera with telephoto lens • system tested in association with saccade and visual tracking • vehicle velocity and yaw rate added to the system via CAN bus to improve tracking of space fixed objects

37. static wide angle camera stabilized camera with telephoto lens Testing of the camera stabilization system "on the road" (2) • three modes were tested: • lane marker was focused in a distance of 60m, followed up to a distance of 20m, then the next was focused on • lane separation was focused on in a constant distance of 40m • nothing was focused on but the camera was stabilized around a constant line of sight • in all modes: bumps could be compensated

38. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

39. Learning from the human brain – another example • a current example: face recognition • excellent recognition abilities by humans under different circumstances, e.g. • illumination • viewing angle • pose • facial expression • ... • need for technical face recognition systems (e.g. war against terrorism) • consequently need to continue exploring the neuroscience of face recognition

40. Face recognition – the problem with different angles of view How can we/a computer tell that each face belongs to the same person?

41. Face recognition – adaption of weights and output as weighted sum testing step: unfamiliar view of the face, but nevertheless the highest output is produced for "Betty" "hidden units": one for each angle of view – the more similar the input is to the "prototype" face of one unit, the more "active" the unit gets learning step: changing the weights within the network so that the output is "1" for the viewed person from: Vatentin, D.; Abdi, H..; Edelman, B.: What represents a face: A Computational Approach for the Integration of Physiological and Psychiological Data, 1997

42. Outline • The human brain • The vestibulo-ocular reflex (VOR) • The VOR – technical applications • Camera stabilizing system at TU Munich • A possible algorithm • Testing the camera stabilizing system • Another field of neuroscience and technical application: face recognition • Conclusion and outlook

43. Conclusion and outlook • technical use of neuroscience is the key to • giving machines several typical human abilities • optimizing service intervals and • minimizing complexity of installing systems by self-learning abilities • cost reduction • ... • consequently interesting to a large variety of technical fields

44. Thank you! Thank you for your attention! Fabian DiewaldFabian.Diewald@mytum.de