Collapsed Variational Dirichlet Process Mixture Models

1. Collapsed Variational Dirichlet Process Mixture Models Kenichi Kurihara, Max Welling and Yee W. Teh Published on IJCAI 07 Discussion led by Qi An

2. Outline • Introduction • Four approximations to DP • Variational Bayesian Inference • Optimal cluster label reordering • Experimental results • Discussion

3. Introduction • DP is suitable for many density estimation and data clustering applications. • Gibbs sampling solution is not efficient enough to scale up to the large scale problems. • Truncated stick-breaking approximation is formulated in the space of explicit, non-exchangeable cluster labels.

4. Introduction • This paper • propose an improved VB algorithm based on integrating out mixture weights • compare the stick-breaking representation against the finite symmetric Dirichlet approximation • maintain optimal ordering of cluster labels in the stick-breaking VB algorithm

5. Approximations to DP • Truncated stick-breaking representation The joint distribution can be expressed as:

6. Approximations to DP • Finite symmetric Dirichlet approximation The joint distribution can be expressed as: The essential difference from TSB representation is that the cluster labels remain interchangeable under this formulation.

7. Dirichlet process is most naturally defined on a partition space while both TSB and FSD are defined over the cluster label space. Moreover, TSB and FSD also live in different spaces

8. Marginalization • In variational Bayesian approximation, we assume a factorized form for the posterior distribution. • However it is not a good assumption since changes in π will have a considerable impact on z. If we can integrate out π, the joint distribution is given by For the TSB representation: For the FSD representation: α

9. VB inference • We can then apply the VB inference on the four approximations The lower bound is given by The approximated posterior distribution for TSB and FSD are Depending on marginalization or not, v and π may be integrated out.

10. Gaussian approximation • For collapsed approximations, the computation for q(zij) seems intractable due to the exponentially large space of assignments for all other {zij}. With central limit theory and Taylor expansion, the expectation over zij will be approximated with those expectations

11. Optimal cluster label reordering • For FSB representation, the prior assumes a certain ordering of the clusters. • The authors claims the optimal relabelling of the clusters is given by ordering the cluster sizes in decreasing order.

12. Experimental results

13. Experimental results

14. Discussion • There is very little difference between variational Bayesian inference in the reordered stick-breaking representation and the finite mixture model with symmetric Dirichlet priors. • Label reordering is important for the stick-breaking representation • Variational approximation are much more efficient computationally than Gibbs sampling, with almost no loss in accuracy