Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations

2026-08-01

AI Summary

Existing methods for evaluating hallucinations in scientific agents overlook the topological structure of knowledge, relying instead on isolated facts and end-task accuracy, which fails to uncover how errors propagate through multi-step reasoning. This work proposes SCHEMA, a novel framework that introduces scientific concept graphs to guide hallucination assessment. By automatically constructing concept graphs to generate graph-anchored tasks and integrating trajectory-based hallucination analysis with multi-agent counterfactual attribution, SCHEMA enables mechanism-level evaluation. Experiments reveal that hallucinations concentrate at highly connected knowledge hubs, and models often reach correct conclusions via structurally flawed reasoning—demonstrating a decoupling between terminal accuracy and reasoning fidelity. These findings advocate shifting reliability evaluation from outcome-oriented metrics toward mechanism-aware assessment.

Abstract

Large language model (LLM) agents are increasingly deployed in scientific research, where reliability is critical and the underlying knowledge is densely interconnected. In such settings, hallucinations are particularly damaging: a single erroneous claim on a foundational concept can propagate through multi-step reasoning and corrupt entire trajectories. Existing hallucination benchmarks largely operate at the surface level, treating facts in isolation and relying on uniform accuracy metrics that ignore this topological structure. We address this gap with SCHEMA, the first evidence-grounded, topology-aware evaluation framework for hallucinations in scientific agents. SCHEMA automatically constructs scientific concept graphs from benchmark seeds and literature evidence, synthesizes graph-grounded tasks spanning claim verification, multi-hop reasoning, open-ended explanation, and experimental code generation, and evaluates agents with two complementary diagnostics. A trajectory hallucination pipeline audits intermediate reasoning at scale via a topology-weighted severity score, while a multi-agent counterfactual attribution module pinpoints the causal mechanism behind selected failures. SCHEMA reveals that hallucinations concentrate at a small set of highly connected knowledge hubs, and that final-answer accuracy decouples from trajectory honesty; models often reach correct conclusions through structurally flawed reasoning. These results indicate that for high-stakes scientific applications, terminal accuracy alone is an insufficient signal of agent reliability, motivating mechanism-level evaluation grounded in knowledge topology. Code is available at https://github.com/circles-post/SCHEMA.

Problem

• hallucination
• scientific agents
• knowledge topology
• reasoning trajectory
• evaluation framework

Innovation

• topology-aware evaluation
• scientific agent hallucination
• concept graph
• trajectory auditing
• counterfactual attribution