Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

2026-10-08

AI Summary

Recurrent Transformers suffer from degraded inference accuracy as iteration depth increases due to accumulated state noise, hindering the correction of early errors. This work proposes InfiLoop, a novel mechanism that introduces the first loop-native attention residual connection, integrating content-aware weighting with learnable temporal decay. Built upon an exact streaming recurrence algorithm, it achieves constant memory overhead while effectively mitigating long-range information loss. Experimental results demonstrate that a model with merely 7M parameters attains 97.9% accuracy on Sudoku-Extreme and 13.6% pass@2 on ARC-AGI-2. Crucially, InfiLoop supports over 20,000 test-time iterations with continuously improving performance, enabling stable reasoning across ten-thousand-scale iterative steps.

Abstract

In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solutions. This leaves subsequent iterations to recover lost information from an already degraded representation: once an error arises in an earlier loop, often as a result of long-range propagation through the recurrence, later loops find it difficult to correct. In this paper, we introduce InfiLoop, a loop-native residual connection that learns which past computations to retain and how much to accept from each new update. InfiLoop combines content-based weighting with learned temporal decay to maintain a running summary of recurrent states. An exact streaming recurrence keeps its persistent aggregation memory constant as the loop count grows. The resulting adaptive update suppresses unreliable proposals and preserves useful intermediate states. Across extensive reasoning tasks, a 7M-parameter InfiLoop model outperforms existing recursive architectures, reaching 97.9% exact accuracy on Sudoku-Extreme, and 13.6% pass@2 on ARC-AGI-2. Notably, on Sudoku-Extreme, InfiLoop continues to improve with test-time looping beyond 20,000 effective steps, showing that added depth translates directly into stronger reasoning. Our code is available at https://github.com/pixeli99/InfiLoop.

Problem

• Looped Transformers
• Test-time scaling
• Reasoning degradation
• Residual connections
• Iterative reasoning

Innovation

• Looped Transformers
• Residual Connections
• Test-Time Compute
• Temporal Decay
• Reasoning