When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs

2026-08-04

AI Summary

Traditional voting mechanisms in causal reasoning often fail due to answer dispersion or repetitive errors. This work proposes CALVER, a training-free symbolic verifier that introduces, for the first time, axiom-level causal validation into large language model (LLM) reasoning selection, enabling identification of graph-structurally valid causal paths without relying on reference answers. CALVER leverages Pearl’s causal criteria—including d-separation, backdoor adjustment, and interventions—combined with text-derived causal graphs and Bayesian networks to achieve millisecond-level scoring efficiency on CPU. Evaluated on the CLEAR benchmark, it attains an accuracy of 42.1%, substantially outperforming majority voting, reward models, LLM judges, and confidence-based baselines (all around 30%), and demonstrates consistent superiority across diverse models, network architectures, and causal graph construction scenarios.

Abstract

Self-consistency assumes the most frequent answer among sampled reasoning traces is the most reliable, but this can fail in causal reasoning: samples often repeat the same confounding error, and votes fragment across multiple valid answers, letting an invalid answer win despite a valid minority trace. We introduce CALVER (Causal Axiom-Level VERification), a training-free symbolic verifier that scores structured traces against Pearl's causal criteria, including -separation, backdoor adjustment, and intervention, and selects the highest-scoring candidate without consulting a reference answer. On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30% on identical frozen pools. Scaling the judge to 72B does not close the gap. In an audited clean-core subset, 11 of 21 graph-valid CALVER selections differ from the benchmark's listed answer while still satisfying the requested predicate. The advantage widens with the sampling budget and reproduces across ten published Bayesian networks, a second model family, and settings where the model must build the graph from text. CALVER also improves thresholded average-treatment-effect decisions against exact ground truth, generalizes to logic under a truth-table checker, and scores each candidate in milliseconds on CPU. CALVER needs only a causal structure, supplied outright or built from the text; wherever that holds, selection can aggregate via causal validity.

Problem

• causal reasoning
• self-consistency
• multiple valid answers
• voting failure
• LLMs

Innovation

• symbolic verification
• causal reasoning
• best-of-K selection
• training-free method
• d-separation