LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger

2026-07-30

AI Summary

This work addresses the limitation of existing multimodal visual question answering agents, which rely solely on answer accuracy and thus fail to verify whether reasoning is grounded in genuine evidence, often succumbing to language priors or error cancellation. To remedy this, the authors propose a provenance-constrained state machine framework that maintains a structured evidence ledger recording tool outputs and restricts subsequent reasoning steps to reference only active entries in the ledger, thereby ensuring fully verifiable and repairable inference. Key innovations include a three-tier grounding protocol, an adaptive dual-path scheduler, and an event-triggered verification-and-repair engine, collectively providing formal guarantees against provenance amplification. The approach effectively identifies and mitigates four subtle failure modes, demonstrating consistent improvements in both answer accuracy and reasoning faithfulness across multiple multimodal reasoning benchmarks and mainstream large language models.

Abstract

Multimodal agents for visual question answering increasingly operate as multi-step trajectories that interleave perception, retrieval, and reasoning, yet evaluation still largely reduces to final-answer accuracy. This aggregate signal cannot tell whether a correct answer was reached through grounded evidence, language priors, or accidental error cancellation. We propose to treat a multimodal agent trajectory as a provenance-constrained state machine: tool outputs are normalized into a Structured Evidence Ledger that serves as the trajectory state, downstream reasoning and decision claims may cite only active ledger entries, grounding is checked at the entity and numeric level, and repair is realized as typed state transitions that cannot introduce content without tool-produced provenance. We instantiate this design as LedgerMind (Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger), augmented by a Three-Layer Grounding Protocol, an Adaptive Dual-Path Dispatcher that matches reasoning depth to question complexity, and an Event-Triggered Verification-and-Repair engine with a formal provenance non-amplification guarantee. We use LedgerMind to target four recurring failure patterns that final-answer accuracy tends to obscure: unsupported intermediate reasoning, citation-backed entity hallucination (Phantom Grounding), over-reasoning on simple queries, and repair-time amplification. Experiments across multiple multimodal reasoning benchmarks and backbone MLLMs show that LedgerMind improves both answer accuracy and trajectory-level faithfulness.

Problem

• multimodal reasoning
• visual question answering
• provenance
• hallucination
• trajectory faithfulness

Innovation

• provenance-constrained reasoning
• structured evidence ledger
• multimodal agentic reasoning
• grounding verification
• trajectory faithfulness