Unsupervised Learning in Graphical Models: Directed and Undirected Approaches

1. Lecture 5 Unsupervised Learning in fully Observed Directed and Undirected Graphical Models

2. Last Time • Last time we saw how to learn the parameters in a Maximum Likelihood setting for labeled data: fully observed supervised learning. • We looked at both the generative (Naive Bayes) and the discriminative (linear & logistic regression) models. • Now we turn to unsupervised learning of general Bayes nets and Markov Random field models. • Directed: very easy and intuitive. • Undirected: very easy for decomposable models. hard for non-decomposable models: IPF.

3. Directed Models • Parameterize by full probability tables. • The log-likelihood is a sum of independent tables if there are no hidden variables. • Use counts m(x) as the relevant statistics of the data. • Constraints enforced using Lagrange multipliers • ML estimates for parameters are simple the appropriate counts

4. Undirected GM • The normalization constant spoils the factorization property. • There is still an easy characterization in terms of counts, however this does not automatically provide the ML estimates of the potentials • When models are decomposable however, we can still write down the solution by inspection. • In all other cases we compute the solution iteratively by means of the “iterative proportional fitting procedure”. • IPF leaves Z invariant, and sets the new marginal on a clique equal to the empirical marginal. • IPF is coordinate ascent on Log-Likelihood! • IPF is guaranteed to converge.