Insect neural networks as a visual collision detection mechanism in automotive situations

1. Insect neural networks as a visual collision detection mechanism in automotive situations Richard Stafford (1), Matthias S. Keil (2), Shigang Yue (1), Jorge Cuadri-Carvajo (2), F. Claire Rind (1) 1) School of Biology, Ridley Building, University of Newcastle upon Tyne, NE1 7RU, UK. 2) Instituto de Microelectronica de Sevilla (IMSE), Centro Nacional de Microelectronica (CNM), Avda. Reina Mercedes, 41012, Sevilla, Spain

2. Structure of the talk • Introduction • Improvements to the LGMD model • Light on and light off pathways • Testing the LGMD model • Methods and Results • Further improvements • Biologically inspired filtering of images by lateral inhibition • Analysing the filtered images using EMD like structures • Conclusions

3. Locusts as collision detectors • The Lobula Giant Movement Detector (LGMD) neuron responds most vigorously when objects of certain speeds and sizes approach, as if on a direct collision course • This has been linked to a predator avoidance, gliding behaviour in flying locusts

4. Predator avoidance caused by the LGMD LGMD Spikes Angular subtense of object

5. Inputs and structure of the LGMD

6. Why use the Locust LGMD to detect automotive collisions? • Evolutionary honed collision avoidance system • Efficient circuit – based on insect neurons • Neural architecture well studied • Responds optimally to imminent collisions • Simulated networks respond in a similar manner to real locust

7. Limitations of existing model(e.g. Rind and Bramwell, 1996; Blanchard et al., 2000) • Simulations only tested in simple closed environment • Model needs to work in real automotive situations • Biology of the LGMD is not fully used – model only responds to change in light

8. Structure of the talk • Introduction • Improvements to the LGMD model • Light on and light off pathways • Testing the LGMD model • Methods and Results • Further improvements • Biologically inspired filtering of images by lateral inhibition • Analysing the filtered images using EMD like structures • Conclusions

9. Model Improvements – Light on and Light off Pathways • Small scale spatial antagonism between the pathways helps eliminate noise caused by vibration etc. • Larger scale antagonism can interfere with collision alerts

10. Model Improvements – Light on and Light off Pathways and Block Sum Cells Block Sum Cells Allow small scale antagonism of pathways only Input Image ‘S’ units – light on ~ light off

11. Location of BSC in model Block sum cells occur here

12. Model Improvements - Block Sum Cells Block sum cells obtain excitation from a 10x10 section of the array of ‘S’ units Excitation (+ve only) is passed to the LGMD from the BSC Light on and Light off excitation from ‘S’ units Block Sum Cells Sum light on (-ve) and light off (+ve) excitation to obtain net excitation LGMD

13. Structure of the talk • Introduction • Improvements to the LGMD model • Light on and light off pathways • Testing the LGMD model • Methods and Results • Further improvements • Biologically inspired filtering of images by lateral inhibition • Analysing the filtered images using EMD like structures • Conclusions

14. Testing the model in automotive situations Input video sequences 8 – 25 Hz Input via frame- grabber of Playstation images 8.3 Hz

15. Detecting collisions • Membrane potential of LGMD is obtained from sum of BSC • If a threshold is exceeded then the LGMD produces spikes • If > 2 spikes in 3 timesteps then collision detected

16. Results: LGMD model Entering Tunnel 0 % Stationary car 100 % General Driving 0 % Moving Car 90 % Driving in close proximity 0 % Head on with moving Car 100 % Translating cars 70 % Results show % of times collision was detected even if no collision occurred

17. Why do translating cars proveproblematic? • Excitation is much higher in the LGMD for translating objects • Locust LGMD ignores translating objects partially due to • differences in mathematics of object approach

18. Structure of the talk • Introduction • Improvements to the LGMD model • Light on and light off pathways • Testing the LGMD model • Methods and Results • Further improvements • Biologically inspired filtering of images by lateral inhibition • Analysing the filtered images using EMD like structures • Conclusions

19. Image Filtering by LGMD network ‘S’ units only excited by objects moving in close proximity to car e.g. Colliding or translating objects Input Image ‘S’ units No threat Threat

20. Analysing the biologically filtered images • Analysing patterns of excitation in ‘S’ or ‘BSC’ layers over time shows: • No or little excitation – no threat. LGMD does not reach threshold • Excitation moving in one direction over time – no threat, translating object. LGMD spikes can be suppressed • Excitation moving in all directions over time – collision threat, object on collision course is expanding in all directions. LGMD spikes and produces collision mitigation response

21. Structure of the talk • Introduction • Improvements to the LGMD model • Light on and light off pathways • Testing the LGMD model • Methods and Results • Further improvements • Biologically inspired filtering of images by lateral inhibition • Analysing the filtered images using EMD like structures • Conclusions

22. Incorporation of simple Elementary Movement Detector like units (EMDs) into the model • EMD like units take input from the Block Sum Cells – simplified visual environment • One detected ‘Right’ movement and one ‘Left’ movement • If membrane potential of ‘left’ EMDs was > 5 x potential of ‘right’ EMDs at time t or time t-1 then LGMD spikes were suppressed for time t, t+1 & t+2

23. Location of EMD like units EMDs Suppression of LGMD spikes BSC

24. Results: LGMD incorporating EMDs Entering Tunnel 0 % Unchanged Stationary car 85 % Was 100 % General Driving 0 % Unchanged Moving Car 80 % Was 100 % Driving in close proximity 0 % Unchanged Head on with moving Car 50 % Was 100 % Translating cars 20 % Was 70 % Results show % of times collision was detected even if no collision occurred

25. Results: LGMD and EMDs • Incorporation of EMDs reduce false collision alerts • Real collision detection was also reduced • EMD model was very simple. Using a more advanced (adaptive) model may improve the responses • Non bio-inspired image analysis could also be used on the biologically filtered ‘S’ units to improve model performance

26. Conclusions • Locust based LGMD model can be used for automotive collision detection • In some situations modifications are needed as the LGMD’s function in automotive situations is quite different to the evolved function in the locust • The biologically filtered image can be analysed to further assess the threat of collision

27. Acknowledgements • Project funded by Future and Emerging Technologies Grant from European Union (LOCUST – IST - 2002-38097) • We would like to thank Marrti Soininen of Volvo Car Corporation for supplying the video footage of car crashes

28. Other Improvements to the LGMD model • On-Off cells look at absolute change in image • Lateral inhibition has a greater potential spread to eliminate more non threatening situations • Spiking threshold of LGMD is self variable to allow a greater range of visually complex scenes to be investigated • Model parameters tuned using a Genetic Algorithm to automotive situations

29. Differences between automotive collisions and predator avoidance in locusts • Locusts respond to small, fast moving predators • Final excitation, just before predator strikes, is much higher • This can be used to distinguish between different object types • Small translating objects produce less excitation than larger objects