Low-Rank Friction for Memory-Efficient Transformer Pretraining

2026-09-24

AI Summary

This study addresses the substantial memory overhead incurred by the friction tensor in the iKFAD optimizer by proposing R-iKFAD. The proposed method leverages rank-1 outer product decomposition combined with momentum statistics to reduce memory consumption by approximately 50%. On the theoretical front, based on continuous-time dynamics analysis, this work provides the first proof of the continuous-time convergence rate for rank-1 factored optimizers without requiring a positive damping condition. Empirical evaluations demonstrate that R-iKFAD achieves performance comparable to or better than iKFAD across multiple model architectures. Ultimately, this approach successfully reconciles low memory footprint with high optimization efficacy, offering a practical and theoretically grounded solution for large-scale training scenarios.

Abstract

iKFAD is a recently proposed optimiser that replaces adaptive learning rates with adaptive friction in the momentum dynamics, yet performs as well as Adam. Its limitation is that the full friction tensor $ξ\in\mathbb{R}^{m\times n}$ carries the same $\mathcal{O}(mn)$ memory overhead per layer as Adam's second-moment buffer. Here we replace iKFAD's friction tensor $ξ$ with a rank-1 outer-product factorisation built from row and column momentum statistics, resulting in Rank-1 iKFAD (R-iKFAD). This reduces the friction memory footprint from $\mathcal{O}(mn)$ to $\mathcal{O}(m+n)$ per layer, which approximately halves iKFAD's total optimiser state. Despite this reduction, R-iKFAD maintains parity in performance with iKFAD: experiments on GPT2-Nano, TinyViT, DistilBERT and GPT2-S confirm that it matches or exceeds iKFAD while nearly halving the memory footprint and remaining comparably robust to hyperparameters. We analyse the continuous-time dynamics in two damping regimes. For linear damping ($γ>0$) we prove exponential convergence under strong convexity. For $γ=0$, the preferred option in our experiments, the friction is generated entirely from past momentum and switches off as the momentum vanishes, so geometric convergence cannot be shown. We nonetheless prove convergence to the minimiser, together with matching upper and lower bounds on the energy: of order $t^{-1}$ when the regularisation scale $ε_{\mathrm{stab}}$ is zero, and of order $t^{-1/2}$ when it is positive. To our knowledge this is the first convergence rate for a rank-1 factored optimiser in continuous time, and the first such result that does not require positive damping.

Problem

• memory efficiency
• optimizer
• friction tensor
• Transformer pretraining
• memory overhead

Innovation

• Rank-1 factorization
• Memory-efficient optimizer
• Adaptive friction
• Continuous-time convergence
• Transformer pretraining