🤖 AI Summary
This work addresses the susceptibility of large language models to hallucinations in empirical reasoning—outputs lacking verifiable evidence or formal guarantees. The authors propose EG-VAR, an architecture that uniquely leverages the Lean 4 formal proof kernel as the sole trusted generator of claims. By integrating tool-certified axioms and a source elevation mechanism, EG-VAR ensures every output is bound to a kernel-verified chain of reasoning and tool invocation; otherwise, it abstains and provides a fully traceable audit trail. Evaluated on a TableBench subset, EG-VAR achieves perfect accuracy (120/120), substantially outperforming a 95% baseline. In counterfactual tests, it maintains 100% source fidelity—significantly higher than competing methods (80–90%)—and exhibits remarkably low semantic formalization error rates of 1.7% (Opus) and 3.3% (Sonnet).
📝 Abstract
Tool access alone does not make LLM empirical reasoning governable: accepted outputs need not descend from attested evidence, and accepted deductions need not hold up under formal scrutiny. We present EG-VAR (Evidence-Grounded Verified Agentic Reasoning), a Lean 4-based tool-calling architecture in which the Lean kernel is the sole minter of Verified claims via tool-attestation axioms and declared source lifts. Every verified output structurally descends from an attested tool call (Thm. 3.1) and a kernel-checked chain of valid inference (Thm. 3.2); residual outputs are honest Abstain with a replayable audit trail. On a subcollection of TableBench numerical reasoning (n=120), EG-VAR attains 120/120 versus a 95% same-tool baseline; on counterfactual stress tests (5 domains x 2 models), EG-VAR stays 100% source-faithful while same-tool drops to 80-90% (no-tool 50-80%). With the LLM as deployment-time formalizer, residual semantic-formalization error is 3.3% on Sonnet and 1.7% on Opus. We position EG-VAR as a technical-governance interface for high-stakes empirical claims: a formal sidecar makes the target proposition, source scope, evidence boundary, proof obligation, and abstention condition auditable, eliminating unsupported Verified outputs today while turning formalization errors, lift and source-authority disputes, ambiguities, and abstentions into explicit audit targets. Over time, typed sidecars in datasets, APIs, public records, and AI-generated documents can amortize this formalization burden into reusable infrastructure.