Irina Rish IBM T.J. Watson Research Center

1. Bregman Divergences in Clustering and Dimensionality ReductionCOMS 6998-4: Learning and Empirical Inference Irina Rish IBM T.J. Watson Research Center Slide credits: Srujana Merugu, Arindam Banerjee, Sameer Agarwal

2. Outline • Intro to Bregman Divergences • Clustering with Bregman Divergences • k-means: quick overview • From Euclidean distance to Bregman divergences • Some rate-distortion theory • Dimensionality Reduction with Bregman Divergences • PCA: quick overview • Probabilistic Interpretation of PCA; exponential family • From Euclidean distance to Bregman divergences • Conclusions

3. Distance (distortion) measures in learning • Euclidean distance – most commonly used • Nearest neighbor, k-means clustering, least squares regression, PCA, distance metric learning, etc • But…is it always an appropriate type of distance? No! • Nominal attributes (e.g. binary) • Distances between distributions • Probabilistic interpretation: • Euclidean distance  Gaussian data • Beyond Gaussian? Exponential family distributions Bregman divergences

8. Squared Euclidean distance is a Bregman divergence

9. Relative entropy (i.e., KL-divergence) is another Bregman divergence

27. Recall Bregman Diverences

32. Now, how about generalizing soft clustering Algorithms using Bregman divergences?

33. (natural parameter)