Differential geometry and the design of optimization algorithms

The problem of parameter estimation by nonlinear least-squares minimization can be recast as a problem in differential geometry by interpreting the set of model predictions as a manifold embedded within the space of data. The minimization problem is then to find the point on the manifold closest to the data. It is well known that problems with large numbers of parameters can be extremely difficult, often because of sloppiness, or an extreme degeneracy in the parameters. Geometrically, this is interpreted as the manifold forming a narrow hyper-ribbon. Standard algorithms fail when they approach the boundaries of the manifold. Differential geometry suggests improvements to standard algorithms to avoid this pitfall. By including a geodesic acceleration correction and allowing uphill moves we dramatically improve the success rate and the convergence speed of the standard Levenberg-Marquardt algorithm.