Information Geometry -- Manifolds of Probability Distributions

1. Algebraic Statistic: Penn State U Introduction to Information GeometryShun-ichi AmariRIKEN Brain Science Institute

2. Information GeometryA Unifying FrameworkStatistical Inference,Convex Analysis,Optimization,Machine learning, Signal Processing,Computer Vision

3. Information Geometry -- Manifolds of Probability Distributions

4. Information Geometry Systems Theory Information Theory Statistics Neural Networks Combinatorics Physics Information Sciences Math. AI Vision Riemannian Manifold Dual Affine Connections Optimization Manifold of Probability Distributions

5. Information Geometry ? Gaussian distributions

6. Manifold of Probability Distributions

7. Invariance Invariant under different representation

8. Two Geometrical Structures Riemannian metric affine connection --- geodesic Fisher information Orthogonality: innner product

9. Tangent space Spanned by scores

10. Riemannian Structure

11. AffineConnection covariant derivative; parallel transport straight line

12. Duality: two affine connections Y X Y X Riemannian geometry:

13. Dual Affine Connections e-geodesic m-geodesic

14. Mathematical structure of -connection : dually coupled

15. Divergence: M Y Z positive-definite

16. Kullback-Leibler Divergence quasi-distance

18. divergence KL-divergence

20. Metric and Connections Induced by Divergence (Eguchi) Riemannian metric affine connections

21. Duality:

22. Dually flat manifoldexponential family; mixture family; x: discrete

23. Manifold with Convex Function : coordinates : convex function negative entropy energy mathematical programming, control systems physics, engineering, vision, economics

24. Riemannian metric and flatness (affine structure) Bregman divergence Flatness (affine) : geodesic (not Levi-Civita)

25. Legendre Transformation one-to-one

26. Two affine coordinate systems : geodesic (e-geodesic) : dual geodesic (m-geodesic) “dually orthogonal”

27. Pythagorean Theorem (dually flat manifold) Euclidean space: self-dual

28. Projection Theorem m-geodesic e-geodesic

29. Projection Theorem Q = m-geodesic projection of P to M Q’ = e-geodesic projection of P to M

30. Information Geometry Dually flat manifold; curved submanifold convex potential functions Euclidean space : self-dual Probability distributions Exponential family : : negentropy

31. Dually flat manifold

32. Two Types of DivergenceInvariant divergence (Chentsov, Csiszar) f-divergence: Fisher- structureFlat divergence (Bregman) – convex functionKL-divergence belongs to both classes: flat and invariant

33. KL-divergence divergence : space of probability distributions invariance dually flat space Flat divergence invariant divergence convex functions Bregman F-divergence Fisher inf metric Alpha connection

34. Space of positive measures : vectors, matrices, arrays f-divergence Bregman divergence α-divergence

35. structure -Entropy-- Tsallis Shannon entropy Generalized log

36. conformal transformation -Fisher information

37. Applications of Information GeometryStatistical InferenceMachine Learning and AIComputer VisionConvex ProgrammingSignal Processing (ICA; Sparse)Information Theory, Systems TheoryQuantum Information Geometry

38. Applications to Statistics curved exponential family: : estimator

39. x : discrete X = {0, 1, …, n}

40. High-Order Asymptotics :Cramér-Rao: linear theory quadratic approximation :

41. Semiparametric Statistical Model y linear relation x mle, least square, total least square

42. Linear Regression: Semiparametrics y x

43. semiparametric Statistical Model

44. Least squares?

45. Fiber Bundle

47. estimating function

48. Parallel Transport

49. Estimating Function

50. Example of estimating functions