DiffuseAgent-MI: Distributionally-Grounded,Tool-Integrated Self-Evolving Agents for Faithful Visual Reasoning

2026-08-01

AI Summary

This work addresses the lack of faithfulness—i.e., misalignment between explanations and actual computations—in tool-augmented vision-language agents for multi-step visual reasoning. To resolve this, the authors propose a self-evolving agent framework that introduces mechanism interpretability from a distributional perspective into visual reasoning for the first time. The approach achieves perceptual grounding via an energy-based model minimizing KL divergence over interpretable feature units, and dynamically refines reasoning trajectories using a trajectory-level faithfulness verifier coupled with an energy-based reconditioning repair branch. This dual design ensures both distribution-level and trajectory-level faithfulness. Experiments demonstrate consistent improvements across GeoQA, SciVis, VQA-v2, and an internal dataset, with accuracy gains up to 5.1 points and more than twofold increases in mutual information–based faithfulness and human-rated explainability consistency.

Abstract

Tool-integrated vision-language agents have made remarkable progress on compositional and multi-step visual reasoning. Yet their outputs frequently exhibit unfaithfulness: the stated reasoning path diverges from the computation that actually produced the answer, undermining reliability in safety-critical applications. We present DiffuseAgent-MI, a self-evolving agent whose perceptual grounding is governed by a KL-minimal energy model over feature units, providing a distributional view of visual mechanistic interpretability. The agent learns an energy landscape that softly constrains generated samples to lie near the native prior conditioned on the chosen interpretable unit, closing the gap between the explanation and the internal representation. A verifier then supplies trajectory-level faithfulness rewards, and a repair branch re-conditions the energy when the verifier flags an unfaithful step. On GeoQA, SciVis, VQA-v2 and an in-house multimodal reasoning set, DiffuseAgent-MI improves accuracy by up to 5.1 points over prior self-evolving agents while more than doubling mutual-information faithfulness and human-interpretability agreement. Our analysis shows the energy term and the verifier are complementary: the former guarantees distributional faithfulness, the latter trajectory-level faithfulness, and only their combination closes both gaps.

Problem

• visual reasoning
• faithfulness
• vision-language agents
• self-evolving agents
• mechanistic interpretability

Innovation

• distributional faithfulness
• energy-based model
• self-evolving agent
• visual reasoning
• mechanistic interpretability