Toric Modification on Machine Learning

1. Toric Modificationon Machine Learning Keisuke Yamazaki & Sumio Watanabe Tokyo Institute of Technology

2. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

3. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

4. What is the generalization error? The true model Predictive model Generalization Error MLE MAP Bayes i.i.d. Learning model Data

5. Algebraic geometry connected to learning theory in the Bayes method. • The formal definition of the generalization error. The true model Predictive model Data Learning model

6. Algebraic geometry connected to learning theory in the Bayes method. • The formal definition of the generalization error. The true model Predictive model Another Kullback divergence : Data Learning model

7. Algebraic geometry connected to learning theory in the Bayes method. • The formal definition of the generalization error. The true model Predictive model Another Kullback divergence : Data Learning model Algebraic Geometry Machine Learning / Learning Theory

8. h(w) The zeta function has an important role for the connection. Analytic func. Zeta function : C-infinity func. with compact support h(w) holomorphic function Algebraic Geometry Machine Learning / Learning Theory

9. The zeta function has an important role for the connection. Zeta function : Kullback divergence Prior distribution holomorphic function Algebraic Geometry Machine Learning / Learning Theory

10. The largest pole of the zeta function determines the generalization error. Asymptotic Bayes generalization error [ Watanabe 2001] Algebraic Geometry Machine Learning / Learning Theory

11. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

12. Resolution of singularities Calculation of the zeta function requires well-formed H(w). : Factorized form : Polynomial form

13. Blow-ups The resolution of singularities with blow-ups is an iterative method.

14. Blow-ups The resolution of singularities with blow-ups is an iterative method. Kullback divergence in ML is complicated and has high dimensional w :

15. Blow-ups The bottleneck is the iterative method. Applicable function Any function Computational cost Large

16. Blow-ups Toric Modification Toric modification is a systematic method for the resolution of singularities. Applicable function Any function Computational cost Large

17. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

18. Toric Modification Newton diagram is a convex hull in the exponent space

19. Toric Modification The borders determine a set of vectors in the dual space.

20. Toric Modification Add some vectors subdividing the spanned area.

21. Toric Modification Selected vectors construct the resolution map.

22. Toric Modification Non-degenerate Kullback divergence • The condition to apply the toric modification to the Kullback divergence For all i :

23. Toric Modification Blow-ups Toric modification is “systematic”. Blow-up The search space will be large. We cannot know how many iterations we need. The number of vectors is limited.

24. Blow-ups Toric Modification Toric modification can be an effective plug-in method. Non-degenerate function Applicable function Any function Computational cost Limited Large

25. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

26. An application to a mixture model • Mixture of binomial distributions The true model: Learning model:

27. The Newton diagram of the mixture

28. The resolution map based on the toric modification

29. The generalization error of the mixture of binomial distributions : Polynomial form : Factorized form The zeta function: : Generalization error

30. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

31. Summary • The Bayesian generalization error is derived on the basis of the zeta function. • Calculation of the coefficients requires the factorized form of the Kullback divergence. • Toric modification is an effective method to find the factorized form. • The error of a binomial mixture is derived as the application.

32. Thank you !!! Toric Modificationon Machine Learning Keisuke Yamazaki & Sumio Watanabe Tokyo Institute of Technology

34. Asymptotic form of G.E. has been derived in regular models. Fisher information matrix Essential assumption: