🤖 AI Summary
This work addresses the limitation of standard LoRA, which employs a uniform rank across all layers despite their varying contributions to task adaptation. The authors propose a lightweight, gradient-variance-based rank allocation strategy: prior to fine-tuning, they perform eight calibration backward passes and use the gradient variance of LoRA-B matrices as a proxy for layer informativeness to proportionally distribute a fixed rank budget. This approach introduces no additional parameters, training overhead, or architectural modifications, while yielding interpretable rank assignments. Furthermore, by employing a simplified diagonal approximation of the empirical Fisher information matrix (eFIM) computed solely over LoRA adapters, memory consumption is reduced by approximately 256×. Experiments demonstrate performance on par with standard LoRA on GLUE (88.6 vs. 88.7) and LLaMA-3-8B commonsense reasoning tasks (68.5 vs. 68.7), confirming its efficacy.
📝 Abstract
Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix - a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calibration backward passes, compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally. The resulting adapter is a standard LoRA with a per-layer rank pattern - no new parameters, no training overhead, no changes to serving infrastructure. We implement this via an efficient approximation of the empirical Fisher Information Matrix (eFIM) diagonal, restricted to LoRA adapter matrices only, which reduces memory cost by approximately 256x compared to full-model Fisher estimation. On GLUE with DeBERTa-v3-base, FIM-LoRA matches LoRA (88.6 vs. 88.7) at the same parameter budget, and on commonsense reasoning with LLaMA-3-8B reaches 68.5 vs. 68.7 for LoRA. The per-layer rank maps are interpretable: value projections and early-to-middle layers consistently receive higher rank, consistent with established findings on transformer layer roles.