Answer with Evidence: Consistency-Aware Grounded Visual Question Answering for Roadside Traffic Scenes

2026-10-04

AI Summary

This study addresses the inconsistency between textual responses and visual evidence grounding in multimodal models applied to roadside traffic scenarios. To mitigate this text-image misalignment, we propose an "enumerate-then-answer" reasoning paradigm that reverses the conventional generation order. Methodologically, reinforcement learning is introduced for policy optimization, employing multi-trajectory ensembles as teacher signals to enable box-free supervised training. Additionally, a novel benchmark and a consistency evaluation metric are constructed. Experimental results demonstrate that the proposed approach improves grounding-response consistency to 93.7% and achieves a localization F1 score of 75.6%, while exhibiting strong generalization capabilities in cross-scenario evaluations.

Abstract

Roadside traffic reasoning requires every free-form textual claim to be backed by visual evidence. Existing grounded multimodal large language models (MLLMs) frequently exhibit say-point mismatch, in which the textual answer contradicts the bounding boxes the model localizes. Evaluation metrics that score answers and boxes separately leave this failure unpenalized. We trace the mismatch to the conventional answer-then-ground factorization, which commits to a numerical claim before any object is enumerated. To measure it, we build RoadSceneVQA-G, a benchmark of 34.7K question-answer pairs in which every free-form answer is linked to the set of boxes that witnesses it, and we propose the Answer-Grounding Consistency (AGC) evaluation suite. To address it, we introduce Enumerate-then-Answer (EtA), which reverses the generation order so that answer-evidence agreement becomes a property of the output structure, and Enumeration-Consistent Policy Optimization (ECPO), a reinforcement learning stage that uses the union of multiple rollouts as a recall teacher without ground-truth boxes. EtA raises say-point consistency from 26.6\% to 93.7\% and grounding F1 from 52.2\% to 73.0\%, and ECPO further increases F1 to 75.6\% without per-box supervision. On gRefCOCO, the same framework outperforms the strongest compared method, indicating that it transfers beyond traffic scenes. The project is available at \url{https://github.com/GuanRunwei/RoadSceneVQA-G}.

Problem

• Visual Question Answering
• Grounded MLLMs
• Say-Point Mismatch
• Answer-Grounding Consistency
• Roadside Traffic Scenes

Innovation

• Enumerate-then-Answer
• Enumeration-Consistent Policy Optimization
• Grounded Visual Question Answering
• Say-Point Consistency
• Reinforcement Learning