Kevin Huang

Understanding scaling laws through random matrix theory

A scaling law is a mathematical description of how a large stochastic system behaves as one or more of its parameters grow. Empirical scaling laws, fitted on measurements of a real-life system, are now widely used to predict training-time performance of LLMs and to monitor inference-time behaviour in AI safety. This talk will take a theoretical angle, and focus on how random matrix theory gives a way to characterise scaling laws. The first half of the talk concerns the joint scaling of depth and width of a neural network, how it maps to analysis of large products of large random matrices, and briefly discuss a recent extension to consider ensembles of these matrix models in the triple scaling regime (arXiv 2607.04047). The second half of the talk concerns the joint scaling of data size and dimension, how random matrix theory explains the “double-descent” phenomenon, and how it informs us on when “multiple-descent” appears instead due to data heterogeneity and dependence — which arises, for example, as a result of data augmentation. This covers a work to appear in the Annals of Statistics and a recent extension (arXiv 2607.24041).