1 / 21

Bregman Information Bottleneck

Koby Crammer Hebrew University of Jerusalem. Noam Slonim Princeton University. Bregman Information Bottleneck. NIPS’03, Whistler December 2003. Motivation. Hello, world. Extend the IB for a broad family of representations

anais
Download Presentation

Bregman Information Bottleneck

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Koby Crammer Hebrew University of Jerusalem Noam Slonim Princeton University Bregman Information Bottleneck NIPS’03, Whistler December 2003

  2. Motivation Hello, world • Extend the IB for a broad family of representations • Relation to the Exponential family Multinomial distribution Vectors

  3. Outline • Rate-Distortion Formulation • Bregman Divergences • Bregman IB • Statistical Interpretation • Summary

  4. Information Bottleneck X T Y X T [p(y=1|X) … p(y=n|X)] [p(y=1|T) … p(y=n|T)]

  5. Rate-Distortion Formulation • Input • Variables • Distortion

  6. Self-Consistent Equations • Bolzman Distribution: • Markov + Bayes • Marginal

  7. Bregman Divergences f:S R Bf(v||u) = f(v) - (f(u)+f’(u)(v-u)) Bf(v||u) = (v,f(v)) f (u,f(u)) (v, f(u)+f’(u)(v-u))

  8. Bregman IB: Rate-Distortion Formulation • Functional • Bregman Function • Input • Variables • Distortion

  9. Self-Consistent Equations • Bolzman Distribution: • Prototypes: convex combination of input vectors • Marginal

  10. Special Cases • Information Bottleneck: • Bregman function: f(x)=x log(x) – x • Domain:Simplex • Divergence:Kullback-Leibler • Soft K-means • Bregman function:f(x)=(1/2) x2 • Domain:Realsn • Divergence:Euclidian Distance • [Still, Bialek, Bottou, NIPS 2003]

  11. Bregman IB Rate-Distortion Exponential Family Bregman Clustering Information Bottleneck

  12. Exponential Family Expectation parameters: Examples (single dimension): Normal Poisson

  13. Exponential Family and Bregman Divergences • Expectation parameters: • Properties :

  14. Illustration

  15. Exponential Family and Bregman Divergences • Expectation parameters: • Properties :

  16. Back to Distributional Clustering • Distortion: • Data vectors and prototypes: expectation parameters • Question: For what exponential distribution we have ? Answer: Poisson

  17. Illustration .8 .2 a b 60 40 a b a a b a a a b a a a Product of Poisson Distributions Pr Multinomial Distribution

  18. Back to Distributional Clustering • Information Bottleneck: • Distributional clustering of Poison distributions • (Soft) k-means: • (Soft) Clustering of Normal distributions

  19. Maximum Likelihood Perspective • Distortion • Input: • Observations • Output • Parameters of Distribution • IB functional: EM [Elidan & Fridman, before]

  20. Back to Self Consistent Equations • Posterior: • Partition Function: Weighted b-norm of the Likelihood • b → ∞ , most likely cluster governs • b→0 , clusters collapse into a single prototype

  21. Summary • Bregman Information Bottleneck • Clustering/Compression for many representations and divergences • Statistical Interpretation • Clustering of distributions from the exponential family • EM like formulation • Current Work: • Algorithms • Characterize distortion measures which also yield Bolzman distributions • General distortion measures

More Related