Technology Category
Machine Learning: Deep Neural Architectures and Foundation ModelsCognitive Modeling & Cognitive Systems: (Computational) Cognitive ArchitecturesNatural Language Processing: (Large) Language Models
How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
🤖 AI Summary
This study investigates how sequence modeling architectures affect the foundational capabilities of pretrained language models, revealing significant degradation in state-based architectures (e.g., RNNs, Mamba) under constrained-domain pretraining and out-of-distribution evaluation. To address this, we propose “full-sequence arbitrary selection capability” as a core architectural design principle and instantiate it via a lightweight Top-1 element/block selection mechanism. Through ablation studies, cross-distribution evaluation, and joint efficiency-capability analysis, we demonstrate that this capability strongly correlates with foundational competencies—including long-range dependency modeling and symbolic reasoning. Crucially, the Top-1 block selection architecture fully restores Transformer-level foundational capabilities with negligible computational overhead. Our work provides both theoretically grounded principles and empirically validated pathways for designing efficient, capable sequence modeling architectures. (149 words)
Technology Category
Application Category
📝 Abstract
Pre-trained language models represented by the Transformer have been proven to possess strong base capabilities, and the representative self-attention mechanism in the Transformer has become a classic in sequence modeling architectures. Different from the work of proposing sequence modeling architecture to improve the efficiency of attention mechanism, this work focuses on the impact of sequence modeling architectures on base capabilities. Specifically, our concern is: How exactly do sequence modeling architectures affect the base capabilities of pre-trained language models? In this work, we first point out that the mixed domain pre-training setting commonly adopted in existing architecture design works fails to adequately reveal the differences in base capabilities among various architectures. To address this, we propose a limited domain pre-training setting with out-of-distribution testing, which successfully uncovers significant differences in base capabilities among architectures at an early stage. Next, we analyze the base capabilities of stateful sequence modeling architectures, and find that they exhibit significant degradation in base capabilities compared to the Transformer. Then, through a series of architecture component analysis, we summarize a key architecture design principle: A sequence modeling architecture need possess full-sequence arbitrary selection capability to avoid degradation in base capabilities. Finally, we empirically validate this principle using an extremely simple Top-1 element selection architecture and further generalize it to a more practical Top-1 chunk selection architecture. Experimental results demonstrate our proposed sequence modeling architecture design principle and suggest that our work can serve as a valuable reference for future architecture improvements and novel designs.
Problem
Research questions and friction points this paper is trying to address.
Impact of sequence modeling architectures on base capabilities
Identifying key design principles to prevent capability degradation
Validating full-sequence arbitrary selection as essential for performance
Innovation
Methods, ideas, or system contributions that make the work stand out.
Limited domain pre-training with OOD testing
Full-sequence arbitrary selection capability principle
Top-1 element and chunk selection validation
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
