MDL

1. MDL Many of the figures are provided by Chris Bishop from his textbook: ”Pattern Recognition and Machine Learning”

2. Overview • Clustering with K-means and a proof of convergence that uses energies. • Clustering with a mixture of Gaussians and a proof of convergence that uses free energies • The MDL view of clustering and the bits-back argument • The MDL justification for incorrect inference.

3. Clustering • We assume that the data was generated from a number of different classes. The aim is to cluster data from the same class together. • How do we decide the number of classes? • Why not put each datapoint into a separate class? • What is the objective function that is optimized by sensible clusterings?

4. Assume the data lives in a Euclidean space. Assume we want k classes. Assume we start with randomly located cluster centers The algorithm alternates between two steps: Assignment step: Assign each datapoint to the closest cluster. Refitting step: Move each cluster center to the center of gravity of the data assigned to it. The k-means algorithm Assignments Refitted means

5. Why K-means converges • Whenever an assignment is changed, the sum squared distances of datapoints from their assigned cluster centers is reduced. • Whenever a cluster center is moved the sum squared distances of the datapoints from their currently assigned cluster centers is reduced. • Test for convergence: If the assignments do not change in the assignment step, we have converged.

6. There is nothing to prevent k-means getting stuck at local minima. We could try many random starting points We could try non-local split-and-merge moves: Simultaneously merge two nearby clusters and split a big cluster into two. Local minima A bad local optimum

7. Soft k-means • Instead of making hard assignments of datapoints to clusters, we can make soft assignments. One cluster may have a responsibility of .7 for a datapoint and another may have a responsibility of .3. • Allows a cluster to use more information about the data in the refitting step. • What happens to our convergence guarantee? • How do we decide on the soft assignments?

8. A generative view of clustering • We need a sensible measure of what it means to cluster the data well. • This makes it possible to judge different methods. • It may make it possible to decide on the number of clusters. • An obvious approach is to imagine that the data was produced by a generative model. • Then we can adjust the parameters of the model to maximize the probability that it would produce exactly the data we observed.

9. The mixture of Gaussians generative model • First pick one of the k Gaussians with a probability that is called its “mixing proportion”. • Then generate a random point from the chosen Gaussian. • The probability of generating the exact data we observed is zero, but we can still try to maximize the probability density. • Adjust the means of the Gaussians • Adjust the variances of the Gaussians on each dimension. • Adjust the mixing proportions of the Gaussians.

10. The EM algorithm alternates between two steps: E-step: Compute the posterior probability that each Gaussian generates each datapoint. M-step: Assuming that the data really was generated this way, change the parameters of each Gaussian to maximize the probability that it would generate the data it is currently responsible for. Fitting a mixture of Gaussians .95 .05 .05 .95 .5 .5 .5 .5

11. The E-step: Computing responsibilities Prior for Gaussian i Posterior for Gaussian i • In order to adjust the parameters, we must first solve the inference problem: Which Gaussian generated each datapoint? • We cannot be sure, so it’s a distribution over all possibilities. • Use Bayes theorem to get posterior probabilities Bayes theorem Mixing proportion Product over all data dimensions

12. The M-step: Computing new mixing proportions • Each Gaussian gets a certain amount of posterior probability for each datapoint. • The optimal mixing proportion to use (given these posterior probabilities) is just the fraction of the data that the Gaussian gets responsibility for. Posterior for Gaussian i Data for training case c Number of training cases

13. More M-step: Computing the new means • We just take the center-of gravity of the data that the Gaussian is responsible for. • Just like in K-means, except the data is weighted by the posterior probability of the Gaussian. • Guaranteed to lie in the convex hull of the data • Could be big initial jump

14. More M-step: Computing the new variances • We fit the variance of each Gaussian, i, on each dimension, d, to the posterior-weighted data • Its more complicated if we use a full-covariance Gaussian that is not aligned with the axes.

15. How do we know that the updates improve things? • Updating each Gaussian definitely improves the probability of generating the data if we generate it from the same Gaussians after the parameter updates. • But we know that the posterior will change after updating the parameters. • A good way to show that this is OK is to show that there is a single function that is improved by both the E-step and the M-step. • The function we need is called Free Energy.

16. Why EM converges • There is a cost function that is reduced by both the E-step and the M-step. Cost = expected energy – entropy • The expected energy term measures how difficult it is to generate each datapoint from the Gaussians it is assigned to. It would be happiest assigning each datapoint to the Gaussian that generates it most easily (as in K-means). • The entropy term encourages “soft” assignments. It would be happiest spreading the assignment probabilities for each datapoint equally between all the Gaussians.

17. The expected energy of a datapoint • The expected energy of datapoint c is the average negative log probability of generating the datapoint • The average is taken using the probabilities of assigning the datapoint to each Gaussian. We can use any probabilities we like. parameters of Gaussian i probability of assigning c to Gaussian i Location of datapoint c data-point Gaussian

18. The entropy term • This term wants the assignment probabilities to be as uniform as possible. • It fights the expected energy term. log probabilities are always negative

19. The E-step chooses the assignment probabilities that minimize the cost function (with the parameters of the Gaussians held fixed) • How do we find assignment probabilities for a datapoint that minimize the cost and sum to 1? • The optimal solution to the trade-off between expected energy and entropy is to make the probabilities be proportional to the exponentiated negative energies: • So using the posterior probabilities as assignment probabilities minimizes the cost function!

20. The M-step chooses the parameters that minimize the cost function (with the assignment probabilities held fixed) • This is easy. We just fit each Gaussian to the data weighted by the assignment probabilities that the Gaussian has for the data. • When you fit a Gaussian to data you are maximizing the log probability of the data given the Gaussian. This is the same as minimizing the energies of the datapoints that the Gaussian is responsible for. • If a Gaussian is assigned a probability of 0.7 for a datapoint the fitting treats it as 0.7 of an observation. • Since both the E-step and the M-step decrease the same cost function, EM converges.

21. The advantage of using F to understand EM • There is clearly no need to use the optimal distribution over hidden configurations. • We can use any distribution that is convenient so long as: • we always update the distribution in a way that improves F • We change the parameters to improve F given the current distribution. • This is very liberating. It allows us to justify all sorts of weird algorithms.

22. An incremental EM algorithm • Partial E-step: Look at a single datapoint, d, and compute the posterior distribution for d. • M-step: Compute the effect on the parameters of changing the posterior for d • Subtract the contribution that d was making with its previous posterior and add the effect it makes with the new posterior. We already have this sum but it includes the oldterm for d

23. The model generates data by picking states for each node using a probability distribution that depends on the values of the node’s parents. The model defines a probability distribution over all the nodes. This can be used to define a distribution over the leaf nodes. Beyond Mixture models:Directed Acyclic Graphical models Hidden cause Visible effect

24. Ways to define the conditional probabilities State configurations of all parents For nodes that have discrete values, we could use conditional probability tables. For nodes that have real values we could let the parents define the parameters of a Gaussian Alternatively we could use a parameterized function. If the nodes have binary states, we could use a sigmoid: states of the node p sums to 1 j i

25. What is easy and what is hard in a DAG? • It is easy to generate an unbiased example at the leaf nodes. • It is typically hard to compute the posterior distribution over all possible configurations of hidden causes. It is also hard to compute the probability of an observed vector. • Given samples from the posterior, it is easy to learn the conditional probabilities that define the model. Hidden cause Visible effect

26. Explaining away • Even if two hidden causes are independent, they can become dependent when we observe an effect that they can both influence. • If we learn that there was an earthquake it reduces the probability that the house jumped because of a truck. -10 -10 truck hits house earthquake 20 20 -20 house jumps

27. An apparently crazy idea • Its hard to learn stochastic generative models that use non-linear distributed representations. This is because its hard to infer (or sample from) the posterior distribution over the hidden variables. • Crazy idea: do inference wrong. • Maybe learning will still work • Can we find an objective function that is: • Easy to optimize because it does not require correct inference. • Easy to justify because it makes a sensible trade-off. • Has deep connections to statistical physics and information theory.

28. Approximate inference • For models that use distributed non-linear representations, it is intractable to compute the exact posterior distribution over hidden configurations. So what happens if we use a tractable approximation to the posterior? • e.g. assume the posterior over hidden configurations for each datavector factorizes into a product of distributions for each separate hidden cause. • If we use this approximation for learning, there is no guarantee that learning will increase the probability that the model would generate the observed data. • But maybe we can find a different and sensible objective function that is guaranteed to improve at each update of the parameters.

29. A trade-off between how well the model fits the data and the accuracy of inference approximating posterior distribution true posterior distribution parameters data This makes it feasible to fit very complicated models, but the approximations that are tractable may be poor. new objective function How well the model fits the data The inaccuracy of inference

31. Two ways to derive F • We can derive variational free energy as the objective function that is minimized by both steps of the Expectation and Maximization algorithm (EM). • We can also derive it by using Minimum Description Length ideas.

32. An MDL approach to clustering sender cluster parameters code for each datapoint data-misfit for each datapoint receiver center of cluster perfectly reconstructed data quantized data

33. How many bits must we send? • Model parameters: • It depends on the priors and how accurately they are sent. • Lets ignore these details for now • Codes: • If all n clusters are equiprobable, log n • This is extremely plausible, but wrong! • We can do it in less bits • This is extremely implausible but right. • Data misfits: • If sender & receiver assume a Gaussian distribution within the cluster, -log[p(d)|cluster] which depends on the squared distance of d from the cluster center.

34. Using a Gaussian agreed distribution • Assume we need to send a value, x, with a quantization width of t • This requires a number of bits that depends on

35. What is the best variance to use? • It is obvious that this is minimized by setting the variance of the Gaussian to be the variance of the residuals.

36. Sending a value assuming a mixture of two equal Gaussians The blue curve is the normalized sum of the two Gaussians. • The point halfway between the two Gaussians should cost –log(p(x)) bits where p(x) is its density under the blue curve. • But in the MDL story the cost should be –log(p(x)) plus one bit to say which Gaussian we are using. • How can we make the MDL story give the right answer? x

37. The bits-back argument data • Consider a datapoint that is equidistant from two cluster centers. • The sender could code it relative to cluster 0 or relative to cluster 1. • Either way, the sender has to send one bit to say which cluster is being used. • It seems like a waste to have to send a bit when you don’t care which cluster you use. • It must be inefficient to have two different ways of encoding the same point. Gaussian 0 Gaussian 1

38. Using another message to make random decisions • Suppose the sender is also trying to communicate another message • The other message is completely independent. • It looks like a random bit stream. • Whenever the sender has to choose between two equally good ways of encoding the data, he uses a bit from the other message to make the decision • After the receiver has losslessly reconstructed the original data, the receiver can pretend to be the sender. • This enables the receiver to figure out the random bit in the other message. • So the original message cost one bit less than we thought because we also communicated a bit from another message.

39. The general case data Gaussian 0 Gaussian 1 Gaussian 2 Bits required to send cluster identity plus data relative to cluster center Probability of picking cluster i Random bits required to pick which cluster

40. What is the best distribution? • The sender and receiver can use any distribution they like • But what distribution minimizes the expected message length • The minimum occurs when we pick codes using a Boltzmann distribution: • This gives the best trade-off between entropy and expected energy. • It is how physics behaves when there is a system that has many alternative configurations each of which has a particular energy (at a temperature of 1).

41. Free Energy Temperature Probability of finding system in configuration i Entropy of distribution over configurations Energy of configuration i The equilibrium free energy of a set of configurations is the energy that a single configuration would have to have to have as much probability as that entire set.

42. A Canadian example • Ice is a more regular and lower energy packing of water molecules than liquid water. • Lets assume all ice configurations have the same energy • But there are vastly more configurations called water.

43. EM as coordinate descent in Free Energy • Think of each different setting of the hidden and visible variables as a “configuration”. The energy of the configuration has two terms: • The log prob of generating the hidden values • The log prob of generating the visible values from the hidden ones • The E-step minimizes F by finding the best distribution over hidden configurations for each data point. • The M-step holds the distribution fixed and minimizes F by changing the parameters that determine the energy of a configuration.

44. Stochastic MDL using the wrong distribution over codes • If we want to communicate the code for a datavector, the most efficient method requires us to pick a code randomly from the posterior distribution over codes. • This is easy if there is only a small number of possible codes. It is also easy if the posterior distribution has a nice form (like a Gaussian or a factored distribution) • But what should we do if the posterior is intractable? • This is typical for non-linear distributed representations. • We do not have to use the most efficient coding scheme! • If we use a suboptimal scheme we will get a bigger description length. • The bigger description length is a bound on the minimal description length. • Minimizing this bound is a sensible thing to do. • So replace the true posterior distribution by a simpler distribution. • This is typically a factored distribution.

45. How many components does a mixture need? • Suppose we want the state of the latent variables to impose about 330 bits of constraint on the visible variables • In a mixture, the latent state consists of a choice of ONE of the components of the mixture. • So we need components • In a later lecture we will see how to fit mixture models with this many components in a few minutes. • This involves a lot of parameter sharing!