When Autoregressive Consistency Hurts Safety Alignment

2026-06-02

AI Summary

Current safety alignment mechanisms in large language models often take effect only at the initial stage of generation and are thus vulnerable to circumvention. This work identifies autoregressive consistency as the core mechanism underlying such fragility: models tend to perpetuate harmful content once a safe prefix has been generated. To address this, we propose an adversarial safety alignment framework that explicitly models the worst-case harmful continuation states and introduces a random worst-case insertion attack alongside a corresponding training strategy. This approach compels the model to maintain robust alignment throughout the entire generation process. Experimental results demonstrate that our method substantially enhances resistance to short harmful fragments and significantly improves overall model safety.

Abstract

Safety alignment in large language models (LLMs) is fragile in part because it is often shallow: fine-tuning mainly reshapes the model's behavior near the first few output tokens. We argue that this phenomenon can be understood through autoregressive consistency, the tendency of next-token prediction to preserve and extend the current response trajectory consistently. By analyzing the learning dynamics of safety alignment, we show that autoregressive consistency can concentrate alignment updates on early tokens, offering a mechanistic explanation for shallow safety alignment. The same mechanism also predicts a broader class of attacks on LLMs: attacks that induce harmful continuation states at arbitrary positions in the output trajectory. As a concrete example, we introduce random insertion attack, which inserts a short harmful span into an otherwise safe refusal trajectory and exploits autoregressive consistency to sustain the resulting harmful branch, thereby bypassing safety alignment. Notably, a short harmful span can redirect the generation to be harmful even after a long refusal prefix, highlighting autoregressive consistency as a potential broader failure mechanism. This suggests that safety alignment should also break harmful autoregressive consistency throughout the output trajectory. We therefore propose adversarial safety alignment, an initial framework based on worst-case harmful continuation states, and instantiate it with random worst-insertion training. Overall, our results suggest that autoregressive consistency should be treated as a central consideration in both safety alignment and attack design.

Problem

• safety alignment
• autoregressive consistency
• harmful continuation
• language model attacks
• shallow alignment

Innovation

• autoregressive consistency
• safety alignment
• random insertion attack
• adversarial safety alignment
• shallow alignment