DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models

2026-06-04

AI Summary

This work addresses the limitations of existing vision-language models, which rely on discrete token-based autoregressive decoding and struggle to generate continuous outputs—such as temporal event boundaries or robotic control actions—efficiently and accurately. To overcome this, the authors propose DRIFT, a novel framework that reformulates global continuous output modeling as learning local residual distributions around strong priors. By integrating a base predictor with a flow-matching-based generative residual refinement module, DRIFT substantially improves both optimization efficiency and prediction accuracy. The approach is highly generalizable and seamlessly integrates with mainstream vision-language architectures, including multimodal large language models (MLLMs), vision-language-action models (VLAs), and world models (WAMs). Experiments demonstrate consistent and significant performance gains over current regression- and generation-based methods across video temporal localization and robotic control tasks, highlighting its strong cross-task and cross-architecture generalization capabilities.

Abstract

Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.

Problem

• vision-language models
• continuous outputs
• temporal localization
• robotic control
• autoregressive decoding

Innovation

• residual flow matching
• continuous output decoding
• vision-language models
• generative refinement
• flow-based generative modeling