Statistics Seminar: Mathematical Foundations for Machine Learning from a Nonlinear Time Series Perspective
Presenter: Jiaqi Li, William H. Kruskal Instructor, University of Chicago
Description:Modern machine learning (ML) algorithms achieve remarkable empirical success, yet providing rigorous statistical guarantees remains a major challenge, particularly in distributional theory and online inference methods. In this talk, we will introduce a novel framework to provide mathematical foundations for ML by bringing powerful tools in nonlinear time series. First, we focus on the stochastic gradient descent (SGD) with constant learning rates. By interpreting the SGD sequence as a nonlinear AR(1) process, we can establish the geometric moment contraction (GMC) for SGD regardless of initializations. By this GMC property, we can derive refined asymptotic theory of SGD and its averaging variant, including general moment convergence, quenched central limit theorems, quenched invariance principles, and sharp Berry- Esseen bounds. Then, we extend this theoretical framework to SGD with dropout regularization, a widely used but theoretically underexplored technique in deep learning. By establishing GMC under explicit learning-rate and dimensional scaling regimes, we obtain asymptotic normality and invariance principles for dropout SGD and its averaged version. These results enable online inference, for which we introduce a fully recursive estimator of the long-run covariance matrix appearing in the limiting distributions. The proposed online confidence intervals with asymptotically correct coverage can be generalized to many other ML algorithms. Overall, viewing online learning algorithms as nonlinear time series provides a powerful toolkit for deriving statistical guarantees in modern ML, with implications for high-dimensional stochastic optimization and real-time uncertainty quantification.
Bio:Jiaqi Li is a William H. Kruskal Instructor in the Department of Statistics at the University of Chicago. She obtained her PhD in Statistics from Washington University in St. Louis in 2024. Her research focuses on developing theoretical guarantees and statistical inference methods for machine learning algorithms. She also works on time series data, especially in the high- dimensional settings with complex temporal and cross-sectional dependency structures. She also
collaborates with neuroscientists on applications in fMRI and EEG data.
Hosted by: Statistics Department
Zoom link: https://ucsc.zoom.us/j/96647674332?pwd=rCHfeGpKslaGS5iIPP5Jh29mQiMJID.1
