1 / 22

Understanding Hopfield Networks: Associative Memory and Energy Functions

This chapter explores Hopfield Networks, a type of neural network composed of McCulloch-Pitts neurons. Learn how to store patterns in the network and retrieve them based on similarity. Discover the energy function and how it affects network dynamics.

weatherly
Download Presentation

Understanding Hopfield Networks: Associative Memory and Energy Functions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Neural NetworksChapter 2 Joost N. Kok Universiteit Leiden

  2. Hopfield Networks • Network of McCulloch-Pitts neurons • Output is 1 iff and is -1 otherwise

  3. Hopfield Networks

  4. Hopfield Networks

  5. Hopfield Networks

  6. Hopfield Networks • Associative Memory Problem:Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing whichever of the stored patterns most closely resembles the new pattern.

  7. Hopfield Networks • Resembles = Hamming distance • Configuration space = all possible states of the network • Stored patterns should be attractors • Basins of attractors

  8. Hopfield Networks • N neurons • Two states: -1 (silent) and 1 (firing) • Fully connected • Symmetric Weights • Thresholds

  9. Hopfield Networks w13 w16 w57 -1 +1

  10. Hopfield Networks • State: • Weights: • Dynamics:

  11. Hopfield Networks • Hebb’s learning rule: • Make connection stronger if neurons have the same state • Make connection weaker if the neurons have a different state

  12. Hopfield Networks neuron 1 synapse neuron 2

  13. Hopfield Networks • Weight betweenneuron i and neuron j is given by

  14. Hopfield Networks • Opposite patterns give the same weights • This implies that they are also stable points of the network • Capacity of Hopfield Networks is limited: 0.14 N

  15. Hopfield Networks • Hopfield defines the energy of a network: E = - ½ ijSiSjwij + i Siqi • If we pick unit i and the firing rule does not change its Si, it will not change E. • If we pick unit i and the firing rule does change its Si, it will decrease E.

  16. Hopfield Networks • Energy function: • Alternative Form: • Updates:

  17. Hopfield Networks

  18. Hopfield Networks • Extension: use stochastic fire rule • Si := +1 with probability g(hi) • Si := -1 with probability 1-g(hi)

  19. 1 g(x) = 1 + e – xb Hopfield Networks • Nonlinear function: b g(x) b 0 x

  20. Hopfield Networks • Replace the binary threshold units by binary stochastic units. • Defineb = 1/T • Use “temperature” T to make it easier to cross energy barriers. • Start at high temperature where its easy to cross energy barriers. • Reduce slowly to low temperature where good states are much more probable than bad ones. A B C

  21. Hopfield Networks • Kick the network our of spurious local minima • Equilibrium: becomes time independent

More Related