Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents

2025-06-02

AI Summary

Nonlinear graph structures and strong image-text coupling in flowcharts induce connection hallucinations in LLMs, severely compromising reliability in critical domains such as logistics and healthcare. To address this, we introduce the fine-grained flowchart attribution task—first of its kind—to trace LLM responses back to specific graph components, thereby enhancing decision verifiability and interpretability. We propose FlowPathAgent, a neuro-symbolic agent that integrates OCR and vision-based segmentation for precise flowchart parsing, employs symbolic graph modeling and graph-traversal-driven agent reasoning to dynamically generate posterior attribution paths. Additionally, we release FlowExplainBench, the first cross-style, cross-domain benchmark for flowchart explanation and attribution. Experiments show our method achieves 10–14% absolute improvement over state-of-the-art baselines on FlowExplainBench, substantially mitigating visual hallucinations while improving structural traceability and domain adaptability of outputs.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Planning, Routing, and Scheduling: Planning with Language Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Flowcharts are a critical tool for visualizing decision-making processes. However, their non-linear structure and complex visual-textual relationships make it challenging to interpret them using LLMs, as vision-language models frequently hallucinate nonexistent connections and decision paths when analyzing these diagrams. This leads to compromised reliability for automated flowchart processing in critical domains such as logistics, health, and engineering. We introduce the task of Fine-grained Flowchart Attribution, which traces specific components grounding a flowchart referring LLM response. Flowchart Attribution ensures the verifiability of LLM predictions and improves explainability by linking generated responses to the flowchart's structure. We propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post hoc attribution through graph-based reasoning. It first segments the flowchart, then converts it into a structured symbolic graph, and then employs an agentic approach to dynamically interact with the graph, to generate attribution paths. Additionally, we present FlowExplainBench, a novel benchmark for evaluating flowchart attributions across diverse styles, domains, and question types. Experimental results show that FlowPathAgent mitigates visual hallucinations in LLM answers over flowchart QA, outperforming strong baselines by 10-14% on our proposed FlowExplainBench dataset.

Problem

• Interpreting non-linear flowcharts with LLMs is error-prone due to visual-textual complexity
• Automated flowchart processing lacks reliability in critical domains like healthcare and logistics
• Existing methods fail to provide verifiable attribution for LLM-generated flowchart responses

Innovation

• Neurosymbolic agent for flowchart attribution
• Graph-based reasoning for verifiable LLM predictions
• Novel benchmark for evaluating flowchart attributions