Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

2026-08-04

AI Summary

This work addresses the high computational cost and misalignment between training objectives and inference distributions in existing reinforcement learning–based methods for aligning vision-language models. To overcome these limitations, the paper introduces a novel test-time alignment paradigm that dynamically optimizes the generation process during inference without requiring expensive post-training. Central to this approach is a trajectory-guided structured sampling mechanism, integrated with an inference memory bank, trajectory learning, and Markov Chain Monte Carlo (MCMC) iterative refinement. This framework significantly enhances both intent alignment and logical consistency in multimodal visual reasoning tasks. Experimental results demonstrate substantial accuracy gains across multiple benchmarks while maintaining low inference overhead, highlighting the method’s efficiency and scalability.

Abstract

Post-training reinforcement learning (RL) algorithms are commonly used to align large vision-language models (LVLMs) with human intent and the requirements of visual reasoning tasks. However, existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions. To bridge this gap, we propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement, achieving better alignment with visual grounding and ensuring logical consistency. Our approach begins with curating a reasoning memory bank via a trajectory learning algorithm, which decomposes complex question solving into ordered sequences of predefined reasoning patterns. It subsequently accomplishes inference-time alignment by first collecting trajectories from reasoning memory bank to establish a global structural reasoning prior, and then using an iterative Markov Chain Monte Carlo (MCMC) algorithm for localized multi-objective refinement of the reasoning trace. Experiments across multiple multimodal reasoning datasets demonstrate that our approach significantly improves accuracy without incurring prohibitive inference overhead. These results establish trajectory-guided test-time sampling as a scalable and effective alternative to traditional post-training alignment, particularly for complex visual reasoning tasks.

Problem

• vision-language models
• test-time alignment
• visual reasoning
• reinforcement learning
• distribution mismatch

Innovation

• test-time alignment
• trajectory-guided sampling
• structured reasoning
• MCMC refinement
• vision-language models