Penelope: Localized Latent Recurrence for Efficient Structured Reasoning

2026-07-28

AI Summary

Current large language models rely on scaling up parameter counts or generating lengthy chains of thought (CoT) for structured reasoning, resulting in high computational costs and latency. This work proposes an efficient latent-variable inference framework that localizes iterative computation to a narrow window within the decoder: it constructs boundary memory through a single prefix pass and employs a temporally modulated GRU for dynamic iterative refinement. Furthermore, a progressive curriculum learning strategy is introduced to transition smoothly from explicit CoT to latent-space reasoning. Without requiring repeated full decoder executions or generating verbose intermediate traces, the method achieves accuracy comparable to existing latent reasoning models across multiple structured reasoning benchmarks while substantially reducing measured inference latency.

Abstract

Complex structured reasoning tasks often require additional computation, yet current language models obtain it mainly by increasing parameter scale or by serializing intermediate steps as chain-of-thought (CoT) tokens. The former raises training and deployment costs, while the latter ties reasoning computation to autoregressive output length. We introduce Penelope, an efficient latent-reasoning framework for pretrained decoder-only Transformers that localizes recurrent computation to a selected decoder interval. The lower decoder prefix is evaluated once to construct a problem-conditioned boundary memory, which is then iteratively refined through time-modulated GRU dynamics and recurrent readout states before answer generation. A progressive CoT-to-latent curriculum transfers visible reasoning into this internal recurrent path, allowing additional computation to be allocated in latent space without repeatedly executing the complete decoder or generating a long intermediate trace. Experiments on open-source structured-reasoning benchmarks show that, at validation-selected latent budgets, Penelope attains competitive accuracy relative to established latent-reasoning models while reducing measured inference latency. These results show that latent refinement can be localized to a narrow decoder interval, reducing repeated full-decoder execution without generating a long visible reasoning trace and providing a practical accuracy-efficiency tradeoff for decoder-only Transformer models.

Problem

• structured reasoning
• latent reasoning
• decoder-only Transformers
• computational efficiency
• chain-of-thought

Innovation

• latent reasoning
• localized recurrence
• decoder-only Transformer
• chain-of-thought
• GRU dynamics