Technology Category
Knowledge Representation and Reasoning: Computational Complexity of ReasoningCognitive Modeling & Cognitive Systems: Conceptual Inference and ReasoningMachine Learning: Large Multimodal Models (LMMs)
Large Language Models Imitate Logical Reasoning, but at what Cost?
🤖 AI Summary
This study systematically evaluates the evolution of logical reasoning capabilities in state-of-the-art large language models (LLMs) over an 18-month period (2023–2025), focusing on accuracy and fidelity to reasoning strategies under in-context learning on the PrOntoQA benchmark. To address computational inefficiency in end-to-end LLM-based reasoning, we propose a lightweight neuro-symbolic architecture: questions are first normalized into first-order logic and then verified for satisfiability using the Z3 SMT solver—enabling near-optimal accuracy with models under 15B parameters. This approach substantially reduces FLOPs and generated token count compared to purely end-to-end methods. Empirical analysis reveals that performance gains from 2023 to 2024 stemmed primarily from implicit chain-of-thought emergence, while advances from 2024 to 2025 were driven by explicit “reasoning models.” Our core contribution is the empirical validation of neuro-symbolic integration as a viable path to high-accuracy logical reasoning with significantly reduced computational overhead.
Technology Category
Application Category
📝 Abstract
We present a longitudinal study which evaluates the reasoning capability of frontier Large Language Models over an eighteen month period. We measured the accuracy of three leading models from December 2023, September 2024 and June 2025 on true or false questions from the PrOntoQA dataset and their faithfulness to reasoning strategies provided through in-context learning. The improvement in performance from 2023 to 2024 can be attributed to hidden Chain of Thought prompting. The introduction of thinking models allowed for significant improvement in model performance between 2024 and 2025.
We then present a neuro-symbolic architecture which uses LLMs of less than 15 billion parameters to translate the problems into a standardised form. We then parse the standardised forms of the problems into a program to be solved by Z3, an SMT solver, to determine the satisfiability of the query. We report the number of prompt and completion tokens as well as the computational cost in FLOPs for open source models. The neuro-symbolic approach significantly reduces the computational cost while maintaining near perfect performance. The common approximation that the number of inference FLOPs is double the product of the active parameters and total tokens was accurate within 10% for all experiments.
Problem
Research questions and friction points this paper is trying to address.
Evaluating reasoning capability of Large Language Models over time
Assessing faithfulness to reasoning strategies in PrOntoQA dataset
Developing neuro-symbolic architecture to reduce computational costs
Innovation
Methods, ideas, or system contributions that make the work stand out.
Hidden Chain of Thought prompting improves reasoning
Thinking models enhance model performance significantly
Neuro-symbolic architecture with Z3 solver reduces cost
🔎 Similar Papers
No similar papers found.
