Beyond the Bidirectional Promise: Re-evaluating the Robustness of Diffusion Language Models

2026-07-29

AI Summary

The robustness and calibration of diffusion language models under natural noise and adversarial attacks remain unclear, posing deployment risks. This work addresses this gap by introducing a parameter-matched paired-model design to disentangle the effects of architecture from those of trained weights, enabling the first systematic comparison with autoregressive models across diverse perturbations. Through comprehensive evaluations—including natural corruption benchmarks, adversarial gradient probing, hidden state analysis, and prompt patching experiments—the study reveals that the apparent robustness of diffusion models stems from their learned weights rather than architectural inductive biases, and further uncovers a pervasive lack of intrinsic noise robustness coupled with systematic overconfidence. Critically, superficial fixes such as prompt patching prove ineffective, underscoring the necessity of embedding robustness mechanisms directly within the iterative decoding process.

Abstract

Diffusion Language Models (DLMs) offer a compelling alternative to autoregressive (AR) generation by enabling bidirectional context and iterative refinement. However, their reliability under natural input noise and adversarial attacks remains under-explored. To address this, we systematically evaluate DLM robustness and calibration against AR baselines, using two parameter-matched pairs (LLaDA-8B vs. LLaMA-3-8B and Dream-7B vs. Qwen2.5-7B) across 32 natural perturbation conditions, adversarial gradient probes, and mechanistic hidden-state analyses. This paired design effectively isolates architecture-intrinsic properties from weight-dependent behaviors. We find a nuanced robustness profile: while highly stochastic DLM loss landscapes naturally resist gradient-based adversarial suffixes, they provide no guaranteed defense against natural noise, proving that everyday robustness is weight-dependent rather than inherently architectural. Furthermore, DLMs exhibit systematic overconfidence, presenting a practical deployment hazard. Most crucially, mechanistic probing reveals that all models perfectly encode input corruption, isolating behavioral fragility entirely to a decoder routing failure. Consistent with this diagnosis, we show that surface-level prompt patching fails to improve over noisy baselines. Ultimately, DLM robustness cannot be patched on; it must be fundamentally integrated into the iterative decoding loop.

Problem

• Diffusion Language Models
• robustness
• adversarial attacks
• natural noise
• decoder routing

Innovation

• Diffusion Language Models
• robustness evaluation
• adversarial attacks
• mechanistic probing
• decoder routing failure