Error whitening: Why Gauss-Newton outperforms Newton

2026-05-11

AI Summary

This work investigates why the Gauss–Newton method often outperforms Newton’s method in practice despite employing only a positive semi-definite approximation of the Hessian. From a function space perspective, the advantage stems from projecting the gradient or Newton direction onto the model’s tangent space, thereby eliminating geometric distortions induced by parameterization. The core contribution is an “error whitening” mechanism that replaces the Jacobian product \(JJ^T\) with the identity matrix, effectively neutralizing parameterization effects and yielding more stable optimization dynamics. Through theoretical analysis grounded in the generalized Gauss–Newton (GGN) matrix, function space projection, and tangent space dynamics, the approach is validated across supervised learning, physics-informed deep learning, and approximate dynamic programming tasks. Empirical results confirm that Gauss–Newton optimizers strictly follow predicted function space trajectories and significantly surpass Newton’s method, Adam, and Muon.

Abstract

The Gauss-Newton matrix is widely viewed as a positive semidefinite approximation of the Hessian, yet mounting empirical evidence shows that Gauss-Newton descent outperforms Newton's method. We adopt a function space perspective to analyze this phenomenon. We show that the generalized Gauss-Newton (GGN) matrix projects the Newton direction in function space onto the model's tangent space, while a Jacobian-only variant obtained by applying the least squares Gauss-Newton matrix to non-least squares losses projects the function space loss gradient onto this same tangent space. Both projections eliminate distortions from the model's parameterization. Specifically, the evolution of the prediction-target mismatch depends on the model's parameterization through the matrix $JJ^\top$ where $J$ is the Jacobian of the model with respect to its parameters. The projections effectively replace $JJ^\top$ with the identity. We call this effect error whitening. Once the parameterization is removed, the prediction-target mismatch evolves according to dynamics dictated by the structure of the loss and the projection produced by the optimizer. Error whitening is a special property of Gauss-Newton descent that rigorously distinguishes it from Newton's method. We empirically demonstrate that Gauss-Newton optimizers follow the theoretically predicted function space dynamics and outperforms Newton's method, Adam, and Muon across case studies spanning supervised learning, physics-informed deep learning, and approximate dynamic programming.

Problem

• Gauss-Newton
• Newton's method
• optimization
• function space
• error whitening

Innovation

• Gauss-Newton
• error whitening
• function space
• parameterization invariance
• tangent space projection