SymboUQ: Symbolic Uncertainty Quantification for Spatial Reasoning in LLMs

2026-07-31

AI Summary

This work addresses the challenge that intermediate relational statements generated by large language models in spatial reasoning tasks often conflict with their final conclusions, rendering token-based confidence scores unreliable for assessing answer correctness. To tackle this issue, the authors propose SymboUQ, a novel framework that decouples symbolic expressibility from semantic certainty for the first time, enabling the construction of an unsupervised certainty profile. SymboUQ further introduces a certainty-aware reliability fusion mechanism, integrating a layout auditor and constraints on both representation and decoding to quantify uncertainty effectively. Evaluated across five spatial reasoning benchmarks, SymboUQ achieves a relative improvement of approximately 8% in AUROC and reduces the class-balanced Brier score by about 7% compared to the strongest baseline.

Abstract

Although large language models (LLMs) can produce fluent spatial reasoning traces, their intermediate relations may fail to support the final conclusion, making token-level confidence insufficient for final-answer reliability estimation. Existing formal verifiers provide stronger semantic evidence, but their applicability is partial: a parsed claim need not yield a definite semantic verdict. To address this issue, we introduce SymboUQ, a symbolic uncertainty quantification framework that estimates final-answer reliability from reasoning traces by distinguishing symbolizability, whether a claim can be represented in the verifier's formal language, from semantic determinacy, whether its execution yields an entailed or contradicted verdict rather than an unknown or not-evaluable outcome. SymboUQ comprises (i) a Layout Auditor that executes ordered spatial claims and extracts feasibility, conflict, and repair evidence; (ii) a label-free Determinacy Profile that characterizes effective executable coverage; and (iii) a Determinacy-Aware Reliability Composer that integrates constraint-based, representation-based, and decoding-based scores according to verifier applicability. Extensive experiments on five spatial reasoning benchmarks with four frozen LLM backbones show that SymboUQ achieves approximately an 8% relative improvement in AUROC and a 7% relative reduction in class-balanced Brier loss over the strongest baseline.

Problem

• uncertainty quantification
• spatial reasoning
• large language models
• semantic determinacy
• symbolizability

Innovation

• Symbolic Uncertainty Quantification
• Spatial Reasoning
• Semantic Determinacy
• Formal Verification
• Large Language Models