1 / 37

Information Geometry of MaxEnt Principle

MaxEnt 07’. Information Geometry of MaxEnt Principle. Shun-ichi Amari RIKEN Brain Science Institute. Information Geometry. Systems Theory. Information Theory. Statistics. Neural Networks. Combinatorics. Physics. Information Sciences. Math. AI. Riemannian Manifold

bishop
Download Presentation

Information Geometry of MaxEnt Principle

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MaxEnt 07’ Information Geometry of MaxEnt Principle Shun-ichi Amari RIKEN Brain Science Institute

  2. Information Geometry Systems Theory Information Theory Statistics Neural Networks Combinatorics Physics Information Sciences Math. AI Riemannian Manifold Dual Affine Connections Manifold of Probability Distributions

  3. Information Geometry ? Riemannian metric Dual affine connections

  4. Manifold of Probability Distributions

  5. Manifold of Probability Distributions

  6. Invariance 1. Invariant under reparameterization 2. Invariant under different representation

  7. Two Structures Riemannian metric—Fisher information Affine connection -- geodesic, straight line how curved is the manifold?

  8. Riemannian Structure

  9. Kullback-Leibler Divergence quasi-distance

  10. KL-divergence and RiemannianStructure relation Fisher information matrix

  11. AffineConnection covariant derivative straight line

  12. Renyi-Tasallis Exponential connection Entropy KL-divergence Mixture connection Levi-Civita (Riemannian)

  13. Affine Connections e-geodesic m-geodesic

  14. Duality Y X Y X Riemannian geometry:

  15. Independent Distributions

  16. Dually flat manifold S = {p(x), x discrete}

  17. Dually Flat Manifold 1. Potential Functions ---convex (Legendre transformation) 2. Divergence KL-divergence 3. Pythagoras Theorem 4. Projection Theorem

  18. Projection Theorem m-geodesic e-geodesic

  19. Applications to Statistics curved exponential family: : estimation : testing

  20. High-Order Asymptotics :Cramér-Rao

  21. Other Applications • Systems theory • Information theory • Neuromanifold • Belief propagation • Boosting (Murata-Eguchi) • Higher-order correlations • Mathematics --- Orlicz space (Pistone, Gracceli) • Physics --- Amari-Nagaoka, Methods of Information Geometry, AMS & Oxford U., 2000 Amari, Differential-Geometrical Methods of Statistics, Springer, 1985 Kass and Vos, Geomtrical Foundations of Asymptotic Inference, Wiley, 1997 Murrey and Rice, Differential Geometry and Statistics, Chapman, 1993

  22. Exponential Family : dually flat Two coordinate systems

  23. Exponential Family example (1) : discrete distributions Negative entropy

  24. example (2) : Gaussian distributions example (3) : AR model

  25. Legendre transformation

  26. Divergence Pythagorean Theorem m-flat e-flat

  27. Divergence and Entropy equi-divergence: equi-entropy

  28. Dual Foliation Pythagorean theorem

  29. Maximum Entropy

  30. Simple Example : independence

  31. Simple example : Gaussian

  32. Time Series

  33. Geometry Potentials

  34. Stochastic Realization

  35. Dual Problem

  36. Rényi-Tsallis entropy Manifold of positive measures m(x)

  37. Entropy (alpha-entropy) is a fundamental quantity---- It is given rise to from a fundamental geometrical structure. KL-divergence is derived therefrom.

More Related