skip connections

This work investigates rank collapse in deep Transformers at initialization, where nonlinearities and matrix multiplications degrade representational capacity and training stability. The authors systematically analyze how components within feedforward blocks influence rank preservation across depth, unifying skip connections and normalization mechanisms under a common framework as gradient-based rank-preserving strategies. They reveal a fundamental distinction between Pre-Norm and Post-Norm architectures in terms of rank dynamics and demonstrate that the two-matrix structure and width expansion are critical for maintaining full-rank Jacobians. Through spectral analysis, Jacobian rank tracking, Marchenko–Pastur law modeling, and CIFAR-10 experiments, they establish that the rank of the input–output Jacobian at initialization strongly predicts training success, offering a new principle for deep architecture design grounded in rank evolution.

To address the dual bottlenecks of shortcut degradation and channel-width limitation in deep residual networks, this paper proposes the Progressive Channel Separation Generalized Residual architecture (PCS-Res). PCS-Res stacks modules that incrementally expand channel dimensions while employing generalized residual connections, thereby dynamically enhancing model capacity and feature propagation efficiency without increasing network depth. It is the first work to systematically identify and jointly mitigate both shortcut degradation and width constraints, breaking the conventional depth-width trade-off. The method is fully compatible with mainstream backbone architectures and supports plug-and-play integration across diverse tasks—including image classification, object detection, semantic segmentation, and language modeling. Extensive experiments demonstrate that PCS-Res consistently outperforms state-of-the-art residual models (e.g., ResNet, ConvNeXt, ViT) across multiple benchmarks, significantly improving representational power and generalization performance of deep networks.

This work addresses the “seesaw effect” — the inherent trade-off between vanishing gradients and representational collapse — in residual connection variants. We propose Hyper-Connection, a novel mechanism built upon learnable gating and depth-aware weight allocation, enabling *joint dynamic control* of connection strength and network topology for the first time. It supports adaptive cross-depth feature routing and on-the-fly layer structural reconfiguration. Theoretical analysis shows it circumvents the fundamental limitations of existing residual variants. Lightweight and plug-and-play, Hyper-Connection requires no modifications to optimizers or training pipelines, and is compatible with both dense and sparse large language models (LLMs) as well as vision tasks. Experiments demonstrate significantly accelerated convergence and improved downstream performance in LLM pretraining; consistent gains are also observed on vision benchmarks, validating its cross-modal generalizability and architectural universality.

Residual identity shortcuts impede generative models’ learning of abstract semantic features, limiting representational capacity and degrading generation quality. This work is the first to identify and characterize this detrimental mechanism. We propose a layer-depth-adaptive identity decay strategy: monotonically reducing the weight of the identity path with increasing network depth to facilitate progressive feature abstraction. Our method is broadly applicable to masked autoencoders (MAEs) and diffusion models. Integrated into ViT-B/16 backbones, it achieves 72.7% linear probe accuracy and boosts k-NN accuracy on ImageNet-1K from 27.4% to 63.9%. It also improves image generation fidelity in diffusion models. Unlike heuristic architectural modifications, our approach offers an interpretable, plug-and-play residual structure optimization paradigm for generative representation learning—requiring no retraining or architectural redesign, yet delivering consistent gains across self-supervised and generative settings.

To address error accumulation and failure in cached inference of diffusion Transformers (DiTs) caused by unstable dynamic feature propagation, this paper proposes a Spectrally Constrained Long Skip Connection (LSC) mechanism: it introduces U-Net–style long skip connections into the DiT architecture for the first time, coupled with spectral norm regularization to ensure dynamic stability during feature propagation; additionally, a static hierarchical caching strategy is designed to enable cross-timestep reuse of deep features. The method preserves generation quality while significantly improving efficiency—accelerating training by 4.4× and speeding up convergence, and accelerating inference by 1.5–2×. Experiments demonstrate that our approach consistently outperforms existing DiT caching methods on both image and video generation tasks, establishing a new paradigm for efficient and stable diffusion modeling.

This work addresses the well-known difficulty of neural networks in approximating high-frequency functions, where conventional residual connections often fail to effectively capture high-frequency patterns. To overcome this limitation, the authors propose a gradient-enhanced residual connection mechanism that explicitly incorporates input gradients into the skip path for the first time. By forming a learnable convex combination of standard residuals and gradient-based residuals, the method adaptively modulates the network’s reliance on high-frequency information. Theoretically, this design enhances sensitivity to input variations. Empirically, the approach significantly outperforms standard residual networks on high-frequency sinusoidal regression tasks and demonstrates consistent gains in single-image super-resolution, while maintaining competitive performance on standard vision benchmarks such as image classification and segmentation.

This work addresses the performance limitations of multimodal large language models (MLLMs) in OCR tasks, which often arise from interference by high-level semantic gradients that degrade fine-grained visual information during deep feature fusion. To mitigate this issue, the authors propose Detached Skip-Links, a mechanism that decouples forward feature reuse from backward gradient propagation, thereby preserving low-level visual features without introducing additional parameters. Furthermore, they introduce the R-Probe evaluation framework, which quantifies pixel-level reconstructability of visual tokens by integrating asymmetric skip connections, gradient truncation, and a shallow decoder initialized from early LLM layers. Experiments demonstrate consistent and significant improvements in OCR performance across diverse Vision Transformer backbones and multimodal benchmarks, with additional gains observed on general-purpose tasks, validated on a dataset comprising 7 million samples.

This work addresses the limitations of standard U-Net’s concatenation-based skip connections, which often induce channel dimension explosion and fail to suppress noise propagation, thereby compromising denoising performance and interpretability. To overcome these issues, the authors propose Additive U-Net, which replaces conventional concatenation with additive skip connections modulated by learnable non-negative gating scalars. This lightweight and interpretable mechanism explicitly controls the contribution of encoder features, effectively avoiding channel expansion while revealing a natural progression of multi-scale features from high- to low-frequency components. Evaluated on the Kodak-17 benchmark, the model achieves competitive PSNR and SSIM across noise levels (σ = 15, 25, 50) and demonstrates strong robustness to variations in network depth and convolutional kernel scheduling.

Deep graph neural networks (GNNs) often suffer from expressive degradation and oversmoothing due to repeated message passing, especially on heterophilous graphs. To address this, we propose an adaptive initial residual connection mechanism that dynamically modulates residual strength at the node level, enhancing deep information propagation while mitigating oversmoothing. Theoretically, we establish the first Dirichlet energy lower bound for residual connections with nonlinear activations, rigorously proving their ability to preserve embedding diversity—unifying the analysis for both static and adaptive residual settings. Our method supports both learnable and heuristic residual strength configurations, with theoretical guidance for optimizing time complexity. Extensive experiments demonstrate that our approach significantly outperforms standard and state-of-the-art GNNs across diverse graph benchmarks, particularly on heterophilous graphs. Notably, the heuristic variant achieves performance comparable to the learnable version, offering superior efficiency and practicality.

Existing methods for unified image restoration under diverse degradations (e.g., noise, blur, adverse weather) rely on complex architectures—such as Mixture-of-Experts (MoE) or diffusion models—and require precise degradation priors, resulting in high computational cost and poor deployability. To address this, we propose SymUNet, a symmetric U-Net that explicitly models and propagates degradation information across scales via scale-aligned feature aggregation and cross-layer feature propagation, eliminating the need for expert modules or diffusion-based priors. Furthermore, we introduce SE-SymUNet, a CLIP-semantic-enhanced variant: its frozen CLIP backbone extracts degradation-relevant semantics, which are injected via lightweight cross-attention. Extensive experiments demonstrate that both SymUNet and SE-SymUNet surpass state-of-the-art methods across multiple benchmarks—achieving superior performance with significantly fewer parameters and faster inference speed. Code is publicly available.