DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models

2026-08-03

AI Summary

This work addresses the redundant computation in diffusion-based vision-language models, which repeatedly utilize all visual evidence at every denoising step. The authors propose a training-free, dynamic visual evidence allocation framework that, for the first time, treats visual evidence as a resource varying across denoising steps. By adaptively allocating the evidence budget per step based on stage-specific conditions and trajectory risk, and employing a hierarchical view structure for efficient conditional injection, the method significantly reduces visual conditioning overhead. Evaluated on LLaDA-V and LaViDa, the approach achieves an average speedup of 1.55× with only a 1.86% performance drop, demonstrating a highly efficient trade-off between computational cost and model accuracy.

Abstract

Diffusion vision-language models (dVLMs) iteratively denoise masked responses while conditioning each denoising step on visual evidence, making visual conditioning a substantial recurring inference cost. Unlike autoregressive decoding, diffusion generation repeatedly revisits the entire response as uncertainty evolves. Our analysis reveals that visual evidence demand is strongly step-dependent, motivating adaptive allocation across denoising steps. Existing inference acceleration methods operate through decoding-side strategies or visual token compression via pruning and merging, but do not explicitly treat visual evidence as a resource whose demand evolves across the diffusion process. Therefore, we present Denoising-Aware Visual Evidence Trajectory Allocation (DAVET), a training-free framework that allocates visual evidence according to the evolving generation state. Starting from a phase-conditioned evidence trajectory, the proposed allocation policy uses operation demand to set an evidence reserve whose allocation at each denoising step is modulated by trajectory risk. DAVET realizes the resulting budgets through a hierarchy of evidence views constructed from a single visual encoding, separating when and how much evidence is needed from how the evidence views are constructed. Evaluated on two representative dVLMs, LLaDA-V and LaViDa, across multiple visual-understanding benchmarks, DAVET achieves an average speedup of 1.55$\times$ with an average relative performance drop of 1.86\%, showing that denoising-aware visual evidence allocation can reduce visual conditioning cost while largely preserving generation quality.

Problem

• diffusion vision-language models
• visual evidence allocation
• denoising steps
• inference acceleration
• visual conditioning cost

Innovation

• diffusion vision-language models
• visual evidence allocation
• denoising-aware
• inference acceleration
• adaptive resource allocation