ementas

Algebraic Geometry and Statistical Learning Theory

On one hand we will introduce the basics of statistical machine learning, and on the other the basics of algebraic geometry and singularity theory, employed by author to study singular statistical models (and all real world models are singular). In the course we read the book [1] of Sumio Watanabe, whose contents are:

  1. [1-47] Introduction. Basic concepts in statistical learning. Statistical models and learning machines. Statistical estimation methods. Four main formulas. Overview. Probability theory.
  2. [48-76] Singularity theory. Polynomials and analytic functions. Algebraic set and analytic set. Singularity. Resolution of singularities. Normal crossing singularities. Manifold.
  3. [77-104] Algebraic geometry. Ring and ideal. Real algebraic set. Singularities and dimension. Real projective space. Blow-up. Examples.
  4. [105-132] Zeta function and singular integral. Schwartz distribution. State density function. Mellin transform. Evaluation of singular integral. Asymptotic expansion and b-function.
  5. [133-157] Empirical processes. Convergence in law. Function-valued analytic functions. Empirical process. Fluctuation of Gaussian processes.
  6. [158-216] Singular learning theory. Standard form of likelihood ratio function. Evidence and stochastic complexity. Bayes and Gibbs estimation. Maximum likelihood and a posteriori.
  7. [217-248] Singular learning machines. Learning coefficient. Three-layered neural networks. Mixture models. Bayesian network. Hidden Markov model. Singular learning process. Bias and variance. Non-analytic learning machines.
  8. [249-276] Singular statistics. Universally optimal learning. Generalized Bayes information criterion. Widely applicable information criteria. Singular hypothesis test. Realization of a posteriori distribution. From regular to singular.

Pre-requisites (citing the author):

This book involves several mathematical fields, for example, singularity theory, algebraic geometry, Schwartz distribution, and empirical processes. However, these mathematical concepts are introduced in each chapter for those who are unfamiliar with them. No specialized mathematical knowledge is necessary to read this book. The only thing the reader needs is a mathematical mind seeking to understand the real world.

Bibliography

  1. Sumio Watanabe, Algebraic Geometry and Statistical Learning Theory, Cambridge Univesity Press, 2009.
  2. Sumio Watanabe, Mathematical Theory of Bayesian Statistics, CRC Press. 2018.
  3. Sumio Watanabe, Statistical Learning Theory 2021, lectures of the course at Tokyo Institute of Technology, available at

http://watanabe-www.math.dis.titech.ac.jp/users/swatanab/s_l_t2.html

  1. Daniel Murfet, Susan Wei, Mingming Gong, Hui Li, Jesse Gell-Redman, Thomas Quella: Deep Learning is Singular, and That’s Good. https://arxiv.org/abs/2010.11560 And open reviews at https://openreview.net/forum?id=8EGmvcCVrmZ. One-sentence Summary: “An invitation to singular learning theory as a theory of deep learning”

Weekly schedule. We will try to follow Watanabe’s lecture series [3]:

  1. Outline
  2. Probability Theory (1)
  3. Probability Theory (2)
  4. Algebraic Geometry
  5. Resolution of Singularities
  6. Zeta function and RLCT (real log-canonical threshold)
  7. Schwartz Distribution
  8. State Density Function
  9. Asymptotic Expansion of Singular Integral
  10. Empirical Process
  11. Free Energy
  12. Generalization Loss
  13. Cross Validation, Information Criterion, and Phase Transition
  14. Application to Statistics
  15. Appendix : Learning Models