Understanding LLM Parameter Update Sparsity through the Lens of Fisher

2026-09-28

AI Summary

This study addresses the prevalent yet mechanistically unclear phenomenon of sparse parameter updates during large language model (LLM) post-training across diverse paradigms. To investigate this, we propose the diagonal Fisher Information Matrix as a unifying perspective, linking update sparsity to online training dynamics through theoretical derivations and empirical analyses in reinforcement learning and online preference optimization. Specifically, we explore the intrinsic relationship between model sensitivity and gradient sparsity. Our findings reveal that low Fisher values fundamentally drive small gradients, establishing the core mechanism underlying sparse updates. Furthermore, we demonstrate that training strategies employing an initial sparse mask can preserve the vast majority of performance gains. Collectively, this work provides a novel framework for understanding and optimizing LLM post-training by elucidating the shared mechanisms governing parameter update sparsity across different alignment paradigms.

Abstract

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training paradigms suggests shared structure in training dynamics. In this paper, we examine this pattern through the diagonal model Fisher, which measures the sensitivity of the model's output distribution to individual parameters and is independent of any particular reward or teacher signal. Theoretically, we show that small diagonal Fisher leads to small expected gradients across a range of training objectives, providing a common explanation for sparse gradient updates. Empirically, we test this connection in RL and OPD. We find that Fisher identifies where gradients are concentrated, and fixed sparse masks selected from the initial Fisher retain a large proportion of the improvement from full training. Finally, we investigate the mechanisms underlying low Fisher in on-policy training. Our results show that high-probability next tokens tend to have similar parameter sensitivities, contributing to low Fisher. Together, these results establish the diagonal model Fisher as a unifying perspective linking update sparsity to on-policy training dynamics in LLM post-training.

Problem

• LLM post-training
• parameter update sparsity
• Fisher information
• training dynamics

Innovation

• Fisher Information
• Parameter Update Sparsity
• Large Language Models
• Post-training Dynamics
• Sparse Masks