Preprint: Singular-limit analysis of gradient descent with noise injection

18 April 2024

Together with Anna Shalova and Mark Peletier, we study the limiting dynamics of a large class of noisy gradient descent systems in the overparameterized regime. In this regime the set of global minimizers of the loss is large, and when initialized in a neighbourhood of this zero-loss set a noisy gradient descent algorithm slowly evolves along this set. In some cases this slow evolution has been related to better generalisation properties. We characterize this evolution for the broad class of noisy gradient descent systems in the limit of small step size.

Noisy gradient descent may continue to move after reaching the zero-loss set Γ. The left-hand panel shows the level curves of a function L:^2→[0,∞), with the zero-level set Γ marked in red. The middle panel shows a gradient-descent evolution, starting at the top, and converging to Γ. The right-hand panel shows an evolution of the noisy gradient descent with L̂(w,η) := L(w+η). — Noisy gradient descent may continue to move after reaching the zero-loss set $\Gamma$ . The left-hand panel shows the level curves of a function $L:\R^2\to[0,\infty)$ , with the zero-level set $\Gamma$ marked in red. The middle panel shows a gradient-descent evolution, starting at the top, and converging to $\Gamma$ . The right-hand panel shows an evolution of the noisy gradient descent with $\hat L(w,\eta) := L(w+\eta)$ .

Now accepted in Journal of Machine Learning Research.

Publication page · arXiv · pdf