A Dynamic Analog Concurrently-Processed Adaptive Chip

1. A Dynamic Analog Concurrently-Processed Adaptive Chip Malcolm Stagg Grade 11

2. A Dynamic Analog Concurrently-Processed Adaptive Chip • Purpose: • To design a reconfigurable analog neural network on a chip • A chip that can learn over time • For any application neural networks are used • Improvement over previous designs: • High density • High routability • On-chip learning • Multiple learning algorithms

3. A Dynamic Analog Concurrently-Processed Adaptive Chip • Neuron and synapse circuits are created • For TSMC 0.35µm CMOS (Complementary Metal Oxide Semiconductor) implementation • Dense layout of transistors for analog arithmetic circuits • High Density • Neurons and Synapses are as small as possible without compromising performance • Supports both Backpropagation and Hebbian learning on each cell

4. A Dynamic Analog Concurrently-Processed Adaptive Chip • Neural networks: • Have been used for years in many applications, from business and finance to science and engineering • Ideally implemented as arrays of electronic cells • Many learning algorithms have been designed • The first electronic neural network was made in 1951, by Marvin Minsky (called “SNARC”) • Carver Mead pioneered chip implementation of neural cells, designing a retina-like neural circuit • Intel created a popular analog neural network chip in 1989 (called “ETANN”)

5. A Dynamic Analog Concurrently-Processed Adaptive Chip • Backpropagation: • Feedback-based learning algorithm (supervised learning) • Requires calculation of the correct output for an input pattern • Will learn to represent the correct output over time • Hebbian: • Non-feedback-based learning algorithm (unsupervised learning) • Calculates synapse weights based on correlation between input patterns • Will learn to identify and classify input patterns • Hopfield: • Advanced memory recall algorithm, using Hebbian learning

6. in out + - BP BP Hebb BPout Hebb BPin A Dynamic Analog Concurrently-Processed Adaptive Chip Neuron Block Diagram

7. A Dynamic Analog Concurrently-Processed Adaptive Chip Neuron Cell Diagram

8. A Dynamic Analog Concurrently-Processed Adaptive Chip • The neuron design: • Supports Backpropagation and Hebbian • Uses an SRAM cell to enable/disable algorithm-dependent circuits • Uses tanh sigmoid circuit for forwards-propagation (range: [-1, 1]) • Derivative as sech2 for backwards propagation (range: [0, 1])

9. in out BP Weight Update Parameters Hebb BPout BP BPin A Dynamic Analog Concurrently-Processed Adaptive Chip Synapse Block Diagram

10. A Dynamic Analog Concurrently-Processed Adaptive Chip Synapse Cell Diagram

11. A Dynamic Analog Concurrently-Processed Adaptive Chip • The synapse design: • Supports Backpropagation and Hebbian • Uses an SRAM cell to enable/disable algorithm-dependent circuits • Uses the Gilbert multiplier cell for forwards- and backwards- propagation • All inputs are ensured to be symmetrical (c+x, c-x) • Differential pair-based circuits are used • Input range is small for good linearity

12. A Dynamic Analog Concurrently-Processed Adaptive Chip • High routability • Multiple routing pathways between neurons and synapses • Using programmable wires and connections • Enables use of advanced learning algorithms • To evolve/change the layout of the network over time • This allows the network to become more dense and more efficient, while maintaining a low synapse count

13. SRAM SRAM Analog Switch Analog Switch A Dynamic Analog Concurrently-Processed Adaptive Chip Programmable Wire Programmable Connection

14. Neuron Synapse Synapse Synapse A Dynamic Analog Concurrently-Processed Adaptive Chip Routing Cell Diagram

15. A Dynamic Analog Concurrently-Processed Adaptive Chip • Circuit Design • Created and modified analog and mixed-signal circuits for each learning/propagation task • Most circuits are implemented using a small number of MOSFET transistors • Transistors are relatively large (5µm length) and interdigitated for good matching • Simulations have been completed for all cells • CMOS layout has been completed for some of the cells

16. A Dynamic Analog Concurrently-Processed Adaptive Chip Current Summation

17. A Dynamic Analog Concurrently-Processed Adaptive Chip Tanh and Sech2 (Sigmoid and Derivative)

18. A Dynamic Analog Concurrently-Processed Adaptive Chip Differential Pairs (for multiplier input)

19. A Dynamic Analog Concurrently-Processed Adaptive Chip Gilbert Multiplier Cell

20. A Dynamic Analog Concurrently-Processed Adaptive Chip • Layout: • Completed for tanh/sech2 • First a stick diagram was planned • Initial layout planned in Microsoft Visio • Drawn in Cadence Virtuoso using unmatched transistors • Drawn in Cadence Virtuoso using matched (interdigitated) transistors • Planned for other cells, but not yet complete

21. A Dynamic Analog Concurrently-Processed Adaptive Chip Initial Virtuoso Layout of tanh (no matching)

22. A Dynamic Analog Concurrently-Processed Adaptive Chip Virtuoso Layout of tanh with interdigitated matching

23. A Dynamic Analog Concurrently-Processed Adaptive Chip Initial Visio layout of tanh (no matching)

24. A Dynamic Analog Concurrently-Processed Adaptive Chip Simulation Graphs Tanh Sigmoid

25. A Dynamic Analog Concurrently-Processed Adaptive Chip • Sigmoid: • Input Common Mode Range: 0.5V – 1.5V • Good accuracy compared to ideal function of tanh

26. A Dynamic Analog Concurrently-Processed Adaptive Chip Simulation Graphs Sech2 Sigmoid

27. A Dynamic Analog Concurrently-Processed Adaptive Chip • Sigmoid Derivative: • Input Common Mode Range: 0.5V – 1.5V • Good accuracy compared to ideal function of sech2

28. A Dynamic Analog Concurrently-Processed Adaptive Chip Simulation Graphs Multiplier

29. A Dynamic Analog Concurrently-Processed Adaptive Chip • Multiplier: • Input Common Mode Range: 2V – 3V • Symmetrical inputs around center point • Good linearity for input swing of ±150mV • Performance is reduced as input range is extended • Pre-processing differential pair is required

30. A Dynamic Analog Concurrently-Processed Adaptive Chip • Results: • A neural network is simulated in C++ • OCR problem, of 5 training sets of randomly ordered numerical characters • 10x10 pixels • 5% noise • Position offset of ±1 pixel horizontally and vertically • Comparing: • 100%-connected network • 20%-connected network • Simulation of a dynamically re-routed network by removing 10% of the synapses over 1000 cycles

31. A Dynamic Analog Concurrently-Processed Adaptive Chip Results 0 to 1000 training cycles

32. A Dynamic Analog Concurrently-Processed Adaptive Chip Results 200 to 1000 training cycles

33. A Dynamic Analog Concurrently-Processed Adaptive Chip Results 800 to 1000 training cycles

34. A Dynamic Analog Concurrently-Processed Adaptive Chip • Conclusions: • The dynamic routing algorithm is the ideal method of optimizing performance in a hardware neural network • Reducing the number of synapses to increase density • The cell circuits perform very well with system-made restrictions on input ranges • Restrictions increase accuracy and linearity • The arrangement of routing cells between neurons and synapses allows the routing algorithm to perform very efficiently

35. A Dynamic Analog Concurrently-Processed Adaptive Chip • Special Thanks To: • Victoria Stagg • My mother, for all of her help and support • Andrew Stagg • My brother, for all of his help and support • Dr. Jim Haslett, University of Calgary • For use of the ATIPS laboratory, to use Cadence Virtuoso • John Carney, Cadence Design • For the demo license of Cadence Orcad Layout • Dr. Vance Tyree, MOSIS Corporation • For agreeing to accept a fabrication proposal under the MOSIS education program • David Wells, Auton Engineering, Ltd. • For printing my trifold