🤖 AI Summary
This work addresses the limited understanding of how implicit biases of optimizers arise during training. Departing from prior analyses focused on the geometry of the solution space, it innovatively shifts attention to the trajectory of parameter updates through the lens of dynamic information allocation. The study introduces a preconditioning exponent \( p \) to characterize the relative distribution of training signals between weight and bias pathways. Using a minimal linear model, it reveals that weight update components preserve input-dependent residual structures, while bias updates capture the mean direction of residuals. The relative strength of these two components is shown to govern both learning dynamics and generalization performance. This framework offers a novel mechanistic perspective on optimizer-induced implicit bias, providing a tunable and interpretable viewpoint for its analysis.
📝 Abstract
Different optimizers have different update biases, but these biases are usually implicit. Existing studies mainly analyze or control such biases from the geometry of the final solution. However, how optimizer bias forms during training still lacks a clear internal mechanism.
This paper proposes an information allocation dynamics perspective. It interprets optimizer implicit bias as the relative allocation of training signals between weight-like and bias-like parameter pathways. This allocation can be described and adjusted by a continuous preconditioning exponent \(p\).
To characterize this mechanism, we first analyze the update contributions of weight and bias to the same residual signal in a minimal linear model. The weight correction term preserves input-dependent residual signals, while the bias correction term preserves the residual mean direction. They therefore correspond to different projection pathways of the residual signal. After substituting the preconditioned update into the residual update equation, the optimizer can change the relative strength of the weight correction term and the bias correction term through different preconditioning factors. Therefore, optimizer implicit bias is not only reflected in the final solution or the global training trajectory. It is also reflected in the relative write-in ratio of training signals across different parameter pathways.
Overall, this paper moves the analysis of optimizer implicit bias from solution-space geometry to update dynamics during training. It reveals that the relative update allocation between weight and bias-like parameters is an important dynamical mechanism that affects parameter trajectories and generalization behavior.