1 / 15

Computing Gradient Vector and Jacobian Matrix in Arbitrarily Connected Neural Networks

Computing Gradient Vector and Jacobian Matrix in Arbitrarily Connected Neural Networks. Author : Bogdan M. Wilamowski , Fellow, IEEE, Nicholas J. Cotton, Okyay Kaynak , Fellow, IEEE, and Günhan Dündar Source : IEEE INDUSTRIAL ELECTRONICS MAGAZINE Date : 2012/3/28 Presenter : 林哲緯.

ossie
Download Presentation

Computing Gradient Vector and Jacobian Matrix in Arbitrarily Connected Neural Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computing Gradient Vector and Jacobian Matrix inArbitrarily Connected Neural Networks Author : Bogdan M. Wilamowski, Fellow, IEEE, Nicholas J. Cotton, OkyayKaynak, Fellow, IEEE, and GünhanDündar Source : IEEE INDUSTRIAL ELECTRONICS MAGAZINE Date : 2012/3/28 Presenter: 林哲緯

  2. Outline • Numerical Analysis Method • Neuron Network Architectures • NBN Algorithm

  3. Minimization problem Newton's method

  4. Minimization problem Steepest descent method http://www.nd.com/NSBook/NEURAL%20AND%20ADAPTIVE%20SYSTEMS14_Adaptive_Linear_Systems.html

  5. Least square problem Gauss–Newton algorithm http://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm

  6. Levenberg–Marquardt algorithm • Levenberg–Marquardt algorithm • Combine the advantages of Gauss–Newton algorithm and Steepest descent method • far off the minimum like Steepest descent method • Close to the minimum like Newton algorithm • It’s find local minimumnot global minimum

  7. Levenberg–Marquardt algorithm • Advantage • Linear • First-order differential • Disadvantage • inverting is not used at all

  8. Outline • Numerical Analysis Method • Neuron Network Architectures • NBN Algorithm

  9. Weight updating rule First-order algorithm Second-order algorithm MLP FCN ACN α : learning constant g : gradient vector J : Jacobian matrix μ : learning parameter I : identity matrix e : error vector

  10. Forward & Backward Computation Forward : 12345, 21345, 12435, or 21435 Backward : 54321, 54312, 53421, or 53412

  11. Jacobian matrix Row : pattern(input)*output Column : weight p = input number no = output number Row = 2*1 = 2 Column = 8 Jacobin size = 2*8

  12. Jacobian matrix

  13. Outline • Numerical Analysis Method • Neuron Network Architectures • NBN Algorithm

  14. Direct Computation of Quasi-Hessian Matrix and Gradient Vector

  15. Conclusion • memory requirement for quasi-Hessian matrix and gradient vector computation is decreased by(P × M) times • can be used arbitrarily connected neural networks • two procedures • Backpropagation process(single output) • Without backpropagation process(multiple outputs)

More Related