One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse

2026-08-03

AI Summary

This work addresses the sudden collapse observed in low-precision Transformer training, which stems from multiple error sources inducing spectral instability in the query-key (QK) attention mechanism. The study identifies the QK pathway as a common failure channel across diverse error types and demonstrates that unified intervention at this locus alone suffices to effectively prevent collapse. Key contributions include establishing the separability between failure origins and the shared QK failure channel, and providing the first evidence that a unified QK correction strategy outperforms source-specific remedies. To this end, the authors propose QK-Guard, a parameter-agnostic QK normalization controller that dynamically activates based on singular-direction monitoring and logit saturation detection. Experiments show that QK-Guard fully suppresses all tested collapse modes within 60,000 training steps, matching the efficacy of permanently enabled QK normalization while all non-QK interventions fail.

Abstract

A bfloat16 transformer can train normally for many steps and then collapse abruptly. Distinct low-precision errors can trigger the same failure, leaving unclear whether each source needs its own repair or one shared route can be blocked. We isolate a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where fp32 accumulation repairs it, and use the fault as an assay for moving controlled errors across sources. Errors placed outside attention still drive the same query-key (QK) spectral runaway, while correcting only QK keeps training stable with the source fault active. This source-channel dissociation shows that fault source is not failure channel. It holds across the tested architectures and scales and reproduces on a second GPU architecture. A causal probe projects each update off the current QK weights' leading three singular directions: the query projection's largest singular value stays at 11.1, whereas removing equal energy elsewhere leaves it at 237. The QK channel therefore drives the early runaway rather than merely tracking it. Entry depends on temporal sign-coherence across steps, not aggregate deviation. QK-Guard closes the channel with a dormant controller that switches on parameter-free QK normalization when attention-logit saturation begins. It contains every tested runaway and matches always-on QK normalization over 60k steps, while non-QK actions at the same trigger fail. The results support intervention at the shared QK locus rather than separate repair at each fault source.

Problem

• low-precision training
• attention collapse
• QK channel
• transformer instability
• numerical error

Innovation

• QK channel
• low-precision training
• attention collapse
• spectral runaway
• QK-Guard