Analog Circuits for Self-organizing Neural Networks Based on Mutual Information

1. Analog Circuits for Self-organizing Neural Networks Based on Mutual Information Janusz Starzyk and Jing Liang School of Electrical Engineering and Computer Science Ohio University Athens, OH 45701 USA

2. Index • Introduction • System description • Analog counter • Mutual information principle • Conclusion

3. Index • Introduction • System description • Analog counter • Mutual information principle • Conclusion

4. Introduction

5. Index • Introduction • System description • Analog counter • Mutual information principle • Conclusion

6. EBE EBE EBE EBE EBE EBE MbyN +/ MbyN +/ MUX - MUX - Threshold Threshold Value Value EBE EBE EBE EBE EBE EBE MbyN +/ MbyN +/ MUX - MUX - Threshold Threshold Value Value EBE EBE EBE EBE EBE EBE MbyN +/ MbyN +/ MUX - MUX - Threshold Threshold Value Value EBE EBE EBE EBE EBE EBE MbyN +/ MbyN +/ MUX - MUX - Threshold Threshold Value Value System Description EBE Threshold Value Fig. 1 System Description

7. Index • Introduction • System description • Analog counter • Circuit structure • Analysis of the operation • Simulation and results • Mutual information principle • Conclusion

8. Vdd M M 3 4 Clk_in M V_out M 2 1 V_ref M C C 5 1 2 Clr Fig. 2 The analogcounter Circuit Structure

9. Vdd M M 3 4 Clk_in M V_out M 2 1 V_ref M C C 5 1 2 Clr Fig. 2 The analogcounter Analysis of the Operation

10. Simulation and Results Fig. 3. Simulation result of analog counter’s linearity and dynamic range.

11. Index • Introduction • System description • Analog counter • Mutual information principle • Mutual information • Approximations of mutual information function • Test data preparation • Simulation results • Conclusion

12. Mutual Information The entropy based mutual information is defined as follows: (4.1) Where (4.2) And (4.3)

13. Approximations of Mutual Information Function Any classification problem can be decomposed into a number of classification problems of two classes: (4.4) (4.5) (4.6)

14. Approximations of Mutual Information Function

15. Approximations of Mutual Information Function Different approximations can be used in order to implement it in analog circuits. Two different approximation are used here. One is the linear approximation: (4.7) The other one is quadratic approximation: (4.8) Their responses with respect to class probability and its compare with original function is shown in Fig. 4.

16. Test Data Preparation In order to compare the final decisions made by using the original mutual information function and its approximations, we randomly generated test data with normal distributions. The mean value and covariance matrix are different for each class date set. To better shown the distribution of data, these data generated with two dimensions, although we only use one of the dimensions. Each class can have different ellipse shape with major axis in different directions.

17. Simulation Results

18. Simulation Results

19. Simulation Results

20. Index • Introduction • System description • Analog counter • Mutual information principle • Conclusion

21. Conclusion We presented a new self-organizing neural network concept based on mutual information. This network is to be implemented in analog circuits in order to get better performance. To implement the most important circuit in this network, the EBE, an analog counter is presented as a building block to perform statistical calculation and simulated. Satisfying results are achieved. Furthermore, different approximation approaches are used to implement the mutual information calculation; the simulation result shows that it is possible to implement it by linear circuits or quadratic circuits.