Connall Garrod

Universal Geometries in Classification

Deep learning systems have achieved remarkable performance across a wide range of tasks, from image classification and language translation to game playing. These successes are often attributed to the large number of parameters in modern networks, which typically places them in the overparameterized regime. Given this overparameterization, together with the diversity of training data and model architectures, one might expect each network to behave as a bespoke object, with little regularity across tasks or architectures. In this talk, I will present evidence to the contrary, describing a range of universal phenomena that emerge in overparameterized neural networks trained to convergence on classification tasks. The discussion will focus on neural collapse, a particularly striking form of geometric simplicity that emerges in learned representations. I will then show how these observations can be understood theoretically through a principled abstraction of the overparameterized limit. This abstraction reveals connections to the theory of linear neural networks, allowing us to use ideas from the implicit bias and spectral dynamics literatures to explain the emergence of these universal geometries.