The non-equilibrium mechanics of learning in neural networks

Artificial neural networks, like the biological circuits that first inspired them, are high-dimensional systems whose trainable parameters interact with one another and with data in complex, nonlinear ways. Despite this complexity, tools from statistical mechanics, stochastic-process theory, and dynamical systems have recently enabled progress in understanding the mechanisms of learning. In this talk, I will describe work from our group, originally motivated by efforts to infer the brain's learning strategies, that connects microscopic details of the learning algorithm to macroscopic observables such as the evolution of network performance during training. Specifically, I will describe a theory that characterizes the noisy dynamics of parameters trained with stochastic gradient descent (SGD)--the workhorse algorithm used to train virtually all neural networks--as Brownian motion in a high-dimensional potential landscape. This perspective provides insights into the unreasonable effectiveness of SGD for efficiently finding solutions that generalize well to new data and, more broadly, recasts learning as a non-equilibrium physical process, opening the door to a quantitative theory of learning in complex adaptive systems, both biological and artificial.