When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

2026-08-04

AI Summary

This work identifies a critical numerical instability in the Attention with Linear Biases (ALiBi) positional encoding, wherein linear bias scaling causes underflow in floating-point precision, leading to excessive zeroing of attention weights and effectively “blinding” attention heads. The study is the first to characterize this failure mechanism, systematically evaluates its impact on long-context modeling, and introduces four training-time mitigation strategies, including a novel log-distance scaling approach. Experiments on a 148M-parameter decoder demonstrate that standard ALiBi remains a strong baseline on needle-in-a-haystack tasks, while the proposed log-scaled distance consistently enhances performance—particularly in passkey retrieval—significantly improving token retrieval capability and offering clear practical guidance for robust ALiBi training.

Abstract

We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.

Problem

• ALiBi
• positional encoding
• floating-point underflow
• attention mechanism
• numerical failure

Innovation

• ALiBi
• positional encoding
• numerical underflow
• attention blindness
• log-scaled distances