Test-Time Training for Modality Order Consistency in Vision-Language Models

2026-07-22

AI Summary

This work reveals that vision-language models exhibit sensitivity to the input order of image and text, leading to semantically irrelevant performance fluctuations. The study identifies, for the first time, that this phenomenon stems from circuit-level failures within the model architecture. To address this, the authors propose a test-time training method that enforces representational consistency between two distinct input orders—image-first and question-first—thereby achieving cross-layer representation alignment and activation repair. This approach substantially narrows the performance gap induced by modality ordering across all evaluation settings, consistently outperforms the original baseline on the stronger image-first branch, and simultaneously enhances model robustness and overall performance.

Abstract

We find that vision-language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and three benchmarks, image first prompting consistently outperforms question-first prompting, revealing a repeatable modality order failure. We use this gap to design an order-consistent test-time training method. Our method substantially closes the modality-order gap across all evaluated settings. Surprisingly, it also yields consistent improvements in the stronger image-first branch over the baseline, hence bootstrapping both orderings toward mutual consistency. Activation patching localizes the ordering failure to a narrow mid-network region where representations diverge sharply between prompt orders. We find that the test-time training method repairs this misalignment across layers. Together, our results identify modality-order sensitivity as a circuit-level failure in VLMs and demonstrate that simple, asymmetric test-time adaptation can effectively mitigate it and even improve performance over the baseline.

Problem

• modality order
• vision-language models
• test-time training
• order consistency
• prompt ordering

Innovation

• test-time training
• modality order consistency
• vision-language models
• activation patching
• representation alignment