CSC 578 Neural Networks and Deep Learning

1. CSC 578Neural Networks and Deep Learning 9. Hopfield Networks, Boltzmann Machines Noriko Tomuro

2. Unsupervised Neural Networks • Hopfield Networks • Concepts • Boltzmann Machines • Concepts • Restricted Boltzmann Machines • Deep Boltzmann Machines Noriko Tomuro

3. 1 Hopfield Network • A Hopfield network is a form of recurrent artificial neural network. It is also one of the oldest neural networks. • There are many variations.  The one presented here is a discrete network which takes bipolar inputs (1 or -1). • Hopfield network stores patterns -- then recovers stored patterns from partial or corrupted patterns. -- Associative Memory. • Hopfield networks also have been applied to combinatorial optimization problems, e.g. Traveling Salesman Problem Noriko Tomuro

5. x1 W21= W12 W31= W13 x3 Overview of Hopfield Network • Weights between units are bi-directional (thus "Feedback" or "Recurrent" network) => A network is a fully-connected network (but no self-loop weights, i.e., wii = 0). • Each unit/node represents a neuron.  And the activation of a neuron is (in the case of binary network; similar to thresholded perceptron) • If xi is 1, a unit is called "active", and if -1, it is called "inactive". • Every neuron functions as both input and output unit. x2 W23= W32

6. A state of a network is defined by the activation of nodes -- <x1,..,xn>. • Given a set of (current) weights, values of nodes are updated asynchronously (parallel relaxation). • Pick a node randomly, and compute the new activation for that node -- a node fires if it becomes 1. • Repeat the procedure until no node changes value. • Then, the network settles in to one of the stable state.

7. x1 x1 x1 x1 -2 -2 -2 -2 1 1 1 1 x3 x3 x3 x3 x2 x2 x2 x2 1 1 1 1 (0) Pattern <-1, -1, -1> presented (1) After activating x2 -1 -1 -1 -1 1 -1 (3) After activating x3 (2) After activating x1 -1 -1 1 1 1 -1 (4), (5), (6) After activating x1 through x3 again, no more state change occurs. Noriko Tomuro

8. It is proven that Hopfield network with asynchronous update will always converge to a stable state. • Depending on the weights and input values, there may be several states to which the network converges. • The change of the network state is essentially a search through the possible state space. • Closeness to a stable state is measured by the notion of energy • j is the connection weight between unit j and unit i. • is the state, si {0,1}. • is the bias of unit i. (- is the activation threshold for the unit). When a node is activated, the change in the energy is always <= 0.

9. So, searching for a stable state is a minimization problem, and the same Gradient Descent can be used to find the minimum. • However, there is the danger of getting stuck in a local minima -- the convergence goes to the closest local minima.

10. Capacity limitation of Hopfield networks:It has been shown that Hopfield networks can only memorize limited numbers of patterns. • A newer result showed the recall accuracy between vectors and nodes was 0.138 (approximately 138 vectors can be recalled from storage for every 1000 nodes) (Hertz et al., 1991). • A Hopfield network with N nodes can store M patterns where • M = 0.15N (for binary network), or • M = N / 2 log2N (for bipolar network)

11. Training for Hopfield networks:Values in the input patterns are bipolar: e.g. <1, -1, 1>Weights are updated incrementally.We basically want the stored patterns to be the stables states. 0. Initialize network weights. 1. Do until no change in the weights occur 2. Initialize delta_w's to be 0.0. 3. For each pattern d in the training set, do 4. Present the pattern to the network. 5. For each node xi, 6. If xi's activation is different from input xi, 7. update the weights connected to xi. And the weight update for node xi is where eta is the learning rate, and xi, xj are the values in the input pattern d.

12. Modern Hopfield networks: • Instead of using the net to store memories, we use it to construct interpretations of sensory input. The input is represented by the visible units, the interpretation is represented by the states of the hidden units, and the badness of the interpretation is represented by the energy. [Video by Geoff Hinton, 2012]

13. Summary: Hopfield networks • suffer from spurious local minima that form on the energy hypersurface. • require the input patterns to be uncorrelated. • are limited in capacity of patterns that can be stored. • are usually fully connected and not stacked.

14. 2 Boltzmann Machines • A Boltzmann machine (also called stochastic Hopfield network with hidden units) is a type of stochastic recurrent neural network (and Markov random field). • Its units produce binary results. Unlike Hopfield nets, Boltzmann machine units are stochastic.[Wikipedia] A graphical representation of an example Boltzmann machine. Each undirected edge represents dependency. In this example there are 3 hidden units and 4 visible units. https://keras.io/getting-started/sequential-model-guide/

15. The global energy in a Boltzmann machine is identical to that of a Hopfield network: where • j is the connection weight between unit j and unit i. • is the state, si {0,1}. • is the bias of unit i. (- is the activation threshold for the unit). • The goal of learning for Boltzmann machine learning algorithm is to maximize the product of the probabilities that the Boltzmann machine assigns to the binary vectors in the training set. • For the connection between the assigned probabilities and the energy (and temperature) consult this [Wikipedia] page.

16. Noriko Tomuro

17. Training of Boltzmann Machines usually use KL-divergence, or log likelihood. The loss function G (for binary vectors) is where • is the distribution over the training set V • is the distribution over the visible (i.e., not hidden) units in the network And we want to maximize G since the probabilities are not negated.

18. Restricted Boltzmann Machines • It was discovered that original Boltzmann Machines stop learning correctly when the machine is scaled up to anything larger than a trivial machine. • Then in 2014, an architecture called the "restricted Boltzmann machine" or "RBM“ was invented. • RBM does not allow intralayer connections between hidden units. This type of architecture was shown to make inference and learning easier.

20. RBM Learning:

21. Deep Restricted Boltzmann Machines

23. RBM vs. Autoencoder • Autoencoder is a simple 3-layer neural network where output units are directly connected back to input units. The task of training is to minimize an error or reconstruction, i.e. find the most efficient compact representation (encoding) for input data. • RBM shares similar idea, but uses stochastic stochastic units with particular (usually binary of Gaussian) distribution. The task of training is to find out how visible random variables are actually connected/related to hidden random variables. https://www.quora.com/What-is-the-difference-between-autoencoders-and-a-restricted-Boltzmann-machine

24. RBM vs. GAN • Generative adversarial networks (GANs) are a class of artificial intelligence algorithms used in unsupervised machine learning, implemented by a system of two neural networks contesting with each other in a zero-sum game framework. https://en.wikipedia.org/wiki/Generative_adversarial_network

25. https://stats.stackexchange.com/questions/338328/restricted-boltzmann-machines-vs-ganhttps://stats.stackexchange.com/questions/338328/restricted-boltzmann-machines-vs-gan