Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision

2026-08-05

AI Summary

This work systematically evaluates the performance trade-offs and out-of-distribution robustness of Recurrent Vision Transformers (Recurrent ViTs) against conventional deep models under parameter or computational constraints. Fixing a single recurrent block architecture, the study compares three aspects on CIFAR-100: accuracy under matched FLOPs or parameter budgets, the impact of neural ODE solver order, and the effect of staged deep supervision on robustness beyond the training distribution. The key contributions include demonstrating that Recurrent ViTs achieve superior accuracy-parameter efficiency under limited parameters, revealing the continuous-time vector field formulation underlying residual recurrent blocks, showing that higher-order ODE solvers introduce architectural bias rather than numerical gains, and proving that staged deep supervision—while not improving nominal accuracy—significantly enhances extrapolation robustness, preventing performance collapse to random-chance levels.

Abstract

Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-block recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block. Rather than proposing a new architecture, we fix a bViT and provide a controlled empirical characterization of three training and inference regimes under a common CIFAR-100 protocol, asking: (i)~when does recurrence beat independently parameterized depth---at matched FLOPs or at matched parameter memory? (ii)~when a residual recurrent block is trained through an ODE solver, does solver order act as numerical refinement or as an architectural bias? and (iii)~what does robustness beyond the training horizon cost in nominal accuracy? We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy--parameter trade-off under memory constraints. Consistent with the standard view of residual networks as Euler discretizations of ODEs, the continuous-time analogue of a residual recurrent block is the state-subtracted vector field $\dot{z}=F_θ(z)-z$; although known in principle, this distinction is easy to violate when the block is wrapped as a black-box vector field, and we qualify the cost at few accuracy points. Because the vector field is learned jointly with the solver, higher-order solvers act as a solver-induced architectural bias rather than a numerical-accuracy improvement, and their gains are not uniform. Finally, stage-wise deep supervision traces an accuracy--robustness frontier: it does not improve nominal accuracy, but degrades gracefully far beyond the training horizon, where naive recurrence collapses to near-random performance.

Problem

• Recurrent Vision Transformers
• Neural ODEs
• Deep Supervision
• Parameter Efficiency
• Robustness

Innovation

• Recurrent Vision Transformers
• Neural ODEs
• Deep Supervision
• Parameter Efficiency
• Robustness-Accuracy Trade-off