LoopICL: Looping a single transformer block to solve tabular tasks

2026-09-28

AI Summary

This study addresses the parameter redundancy and high inference costs of tabular foundation models by proposing a lightweight architecture based on a recurrent Transformer. Methodologically, it employs a single-block recurrent design that decouples parameter count from network depth to enable efficient weight sharing. A dual-stream attention mechanism, encompassing intra-column and cross-column interactions, is introduced to precisely model tabular structures. Furthermore, by integrating in-context learning with a dynamic early-exit gating strategy, the architecture supports adaptive adjustment of inference depth. Experimental results demonstrate that, under equivalent computational budgets, the proposed approach reduces parameters by 90% while achieving performance comparable to TabICLv2, thereby enabling a flexible trade-off between inference cost and predictive accuracy.

Abstract

Tabular foundation models using in-context learning have recently surpassed gradient-boosted trees on predictive tabular tasks. However, recent mechanistic insights suggest that parameters in these models are largely redundant. We introduce LoopICL, a looped transformer whose core design decouples parameter count from computational depth. LoopICL consists of a single block, processing data through two coupled streams: a cell stream capturing per-cell feature representations and a row stream capturing in-context example representations, jointly refined through within-column and cross-column attention. During pre-training, we vary loop counts, allowing the block to be unrolled for a varying number of iterations at test-time and use a learned exit-gate to automatically exit. In its standard setting, LoopICL performs competitively with TabICLv2 on TabArena and TALENT at the same computational cost (FLOPs), while using nearly 90% fewer parameters. Furthermore, its recurrent design enables users to also trade off inference cost and performance, providing a resource-aware TFM.

Problem

• tabular foundation models
• in-context learning
• parameter redundancy
• computational depth

Innovation

• Looped Transformer
• In-Context Learning
• Tabular Foundation Model
• Parameter Efficiency
• Early Exit