DepthBench: Measuring How Residual Connections Enable More Computational Depth

2026-09-26

AI Summary

This study addresses the diminishing computational efficiency in deep Transformer layers and the unclear impact of residual connections on effective depth. By constructing controlled benchmarks with fixed parameter budgets, we systematically evaluate how different width-to-depth ratios utilize computational depth. This work is the first to quantify the contribution of residual designs in translating nominal depth into effective computation while rigorously excluding confounding factors. Through controlled experiments and layer-wise analysis comparing Pre-LN, HC, and AttnRes architectures, we find that HC and Full AttnRes significantly enhance deep-layer utilization. These results establish residual design as a critical mechanism for depth scaling, providing a theoretical foundation for efficient architecture design.

Abstract

Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (\text{e.g.}, LayerNorm Scaling) or residual connections (\text{e.g.}, mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce \textbf{DepthBench}, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width--depth aspect ratio ($d_{\text{model}}/n_{\text{layer}}$) from shallow--wide to deep--narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower, whereas HC and Full AttnRes improve consistently even at extreme deep shapes. These gains extend beyond pre-training loss and consistently translate into improved domain-specific performance and effective computation. Controlled layer-level analyses further show that the gains of HC and Full AttnRes are associated with more effective utilization of additional layers, revealing distinct mechanisms of computational depth across architectures. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis by enabling additional architectural depth to translate into effective computation.

Problem

• computational depth
• residual connections
• Transformers
• depth scaling
• DepthBench

Innovation

• DepthBench
• Computational Depth
• Residual Connections
• Width-Depth Aspect Ratio
• Transformer Architecture