A Causality-aware Infer-diagnose-refine Framework for Test-time Modality Adaptation in VLA Models

2026-07-28

AI Summary

This work addresses the challenge of multimodal fusion in vision-language-action (VLA) models, where the importance of visual inputs dynamically shifts across different stages of robotic manipulation. To tackle this, the authors propose an Inference-Diagnosis-Refinement (IDR) framework that operates at test time without requiring retraining. IDR infers actions using both factual and counterfactual visual inputs, where counterfactual predictions are generated via zero-filling interventions. By quantifying the causal effect of visual observations through norm-based metrics, the method dynamically estimates visual modality importance and adaptively refines action outputs via a gated residual fusion module. As the first approach to incorporate causal awareness for test-time, adaptive modality weighting, IDR significantly enhances the performance of diverse VLA backbone models on both simulated and real-world robotic tasks.

Abstract

Vision-language-action (VLA) models predict sequential actions to execute tasks specified by language instructions, conditioned on visual observations and proprioceptive states. However, how to fuse modalities in VLA models remains an open problem, since robot manipulation involves dynamic phases, such as long-distance movements and close-range interactions, in which the importance of visual observations may vary over time. In this paper, we propose an infer-diagnose-refine (IDR) framework, a model-agnostic framework that can be integrated with diverse VLA architectures for refining action predictions at test time. IDR first infers actions under factual and counterfactual scenarios of visual observations, and then diagnoses the causal effects of visual observations as the estimated dynamic importance, which is finally used to refine the action predictions in a training-free manner. We further design a causality-aware action refiner to realize the IDR framework, including zero-padding interventions for inferring counterfactual actions, norm-based quantification for diagnosing causal effects, and gated residual fusion for refining actions. Extensive experiments on both simulation benchmarks and real-world tasks show improvements in overall performance across multiple VLA backbones, demonstrating the efficacy of dynamically adjusting visual importance at test time.

Problem

• test-time adaptation
• modality fusion
• vision-language-action models
• causal inference
• dynamic importance

Innovation

• test-time adaptation
• causality-aware
• vision-language-action
• counterfactual reasoning
• modality fusion