Hyper-Connections

2024-09-29

arXiv.org

AI Summary

This work addresses the “seesaw effect” — the inherent trade-off between vanishing gradients and representational collapse — in residual connection variants. We propose Hyper-Connection, a novel mechanism built upon learnable gating and depth-aware weight allocation, enabling *joint dynamic control* of connection strength and network topology for the first time. It supports adaptive cross-depth feature routing and on-the-fly layer structural reconfiguration. Theoretical analysis shows it circumvents the fundamental limitations of existing residual variants. Lightweight and plug-and-play, Hyper-Connection requires no modifications to optimizers or training pipelines, and is compatible with both dense and sparse large language models (LLMs) as well as vision tasks. Experiments demonstrate significantly accelerated convergence and improved downstream performance in LLM pretraining; consistent gains are also observed on vision benchmarks, validating its cross-modal generalizability and architectural universality.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and sparse models, where hyper-connections show significant performance improvements over residual connections. Additional experiments conducted on vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across a wide range of AI problems.

Problem

• Addresses gradient vanishing and representation collapse in neural networks.
• Enables dynamic adjustment of feature connection strengths across layers.
• Improves performance in large language models and vision tasks.

Innovation

• Hyper-connections replace residual connections effectively.
• Adjusts connection strength dynamically between network layers.
• Improves performance in language and vision tasks.