Robust Neural Networks using Motes: Innovating Fault-Tolerant Systems

1. Robust Neural Networks using Motes James Hereford, Tüze Kuyucu Department of Physics and Engineering Murray State University

2. Outline • Introduction • Goal • System overview • Background information • Results • Neural net with independent nodes (motes) • Training of distributed neural net • Demonstration of fault recovery EH2005

3. Why interested • Ultimate goal: • Build devices that never fail • Applications: • NASA: Long journeys (e.g., roundtrip to Mars), space probes • Hazardous/dangerous operations – difficult to repair by humans EH2005

4. System Overview One approach to fault tolerance – redundant components Multiple, redundant sensors T1 Processing circuit T2 Tavg T3 • Steps: • Derive/evolve circuit to average N sensors. • Detect a failure. • Re-evolve to average remaining sensors. Simple processing nodes provide redundancy which opens the possibility of fault tolerance in the processing circuit EH2005

5. System Overview - neural net • Neural net: Each artificial neuron (node) receives weighted info from previous layer, sums data, then passes it to next layer. • Neural nets are trained with an iterative technique that determines the interconnection weights. • Challenge: reprogram the neural net when it is unknown a priori which node failed. Idea: If one of our processing units fails, re-train the whole network using evolvable programming techniques. EH2005

6. System Overview 2 key questions: • How to build the neural net? What hardware device to use for each of the nodes? • How to program the neural net? We need a programming technique that does not require a priori information. EH2005

7. System Overview Hardware components T1 Multiple, redundant sensors T2 Tavg T3 • Required node characteristics • Multiply/add • Memory (Store weights) • Internode communication • Power • Mote characteristics • Processor • Memory • Transmit/receive (wireless) • Power • Interface to sensor boards • Software infrastructure (TinyOS, nesC) Devices? EH2005

8. Background information The function of each node is performed by a mote. Mica2Dot mote Mica2 mote Practicalities: Crossbow, $125 (Mica2), range = 10’s of meters, event driven OS, 433 MHz/900 MHz/2.4 Ghz, programming is non-intuitive! EH2005

9. Background information - Particle Swarm Optimization Use PSO to train and re-train neural net 40 39 . . . . 20 . . . . 4 3 2 1 0 Corn Field 2-D Search Space 0 1 2 3 4 ……………………...20…………………………..39 40 EH2005

10. Background information - PSO • 2 update equations: • Velocity • vn+1 = vn + c1*rand*(pbestn – pn) + c2*rand*(gbestn–pn) • Position • pn+1 = pn + vn+1 • Advantages for our application: • Simple update equations • No “hard” functions (e.g., sqrt, sin, fft) • Can “tune” algorithm via constants c1 and c2 EH2005

11. Results • Results in 3 major areas: • Training of neural network with PSO (simulation) • Fault recovery (simulation + hardware) • Building neural net with independent (physically distinct) nodes EH2005

12. Neural Net – Training Results Comparison of classical NN training techniques vs PSO 2 layer 3 layer EH2005

13. NAND TRAIN RETRAIN XOR Neural Net – Training Results Used PSO to re-train the neural net for a different operation Successful in all cases! EH2005

14. Neural Net – Training Results Failure recovery – failure in hidden node(s) 2x4x1 Showed fault recovery for NAND and XOR operations. Successful in all cases-again! Concern: Highly variable number of evaluations to reprogram. Extreme case showed 200:1 variation. 2x3x1 2x2x1 EH2005

15. Neural Net - Hardware • Built hardware NN out of motes • 2 layer (no hidden layer) neural net • Training times shorter with PSO than with perceptron • Demonstrated fault recovery EH2005

16. Neural Net - Hardware 2 layer neural net with fault recovery EH2005

17. Neural Net - Hardware 3 layer neural net built using motes • Programmed successfully “off-line” • Weights from simulation stored on motes worked fine. EH2005

18. Neural Net - Hardware • Embedded programming • Programmed using base mote as “master” • All training done by output (base) node and weight updates sent to hidden layer nodes • Developing distributed training method • Experiments on-going. Mote to mote “feedback” communications problematic. Once programmed – system is able to withstand the hammer test. EH2005

19. Acknowledgements • David Gwaltney – NASA-MSFC • James Humes • Funding: • Murray State: Committee on Institutional Studies and Research • KY NASA EPSCOR program EH2005

20. Conclusions • Simulated and built neural network with independent processors used for each node • Trained (and re-trained) neural net with PSO • Used motes to do processing, not just sensing • Failure recovery approach – every node is identical and replaceable EH2005