prefix tuning

This work identifies and systematically characterizes a prevalent phenomenon in pre-trained Transformer language models—abrupt angular shifts between the final-layer hidden states and input representations—which leads to imbalanced utilization of intermediate layers. To address this issue, the authors propose Jump Regularization (JREG), a lightweight, architecture-agnostic regularization technique that effectively constrains the magnitude of such angular jumps during pre-training. Experimental results across three Llama model scales demonstrate consistent performance improvements over baseline models when JREG is incorporated, confirming its efficacy and generalizability in promoting more balanced layer-wise representation learning.

This work addresses the weak generalization, poor robustness, and low parameter efficiency of Transformers by systematically introducing optimal control theory into their modeling and training—first of its kind. We formulate a continuous-time dynamical framework that integrates variational inference, dynamics-based regularization, and lightweight controller embedding, enabling theoretically grounded training optimization and architecture design. Our approach departs from conventional black-box hyperparameter tuning, offering an interpretable and analyzable modeling paradigm. Empirical evaluation demonstrates a 46% reduction in training loss with 42% fewer parameters on nanoGPT; a 5.6% loss reduction on GPT-2; and consistent performance gains across diverse tasks—including text generation, sentiment analysis, image classification, and point cloud classification—validating its universality, strong generalization, and robustness.

The functional mechanisms underlying layer-wise operations in Transformer models remain poorly understood, particularly regarding the necessity and interchangeability of layer ordering. Method: This work proposes an empirical analysis framework based on freezing large language models (LLMs) and systematically conducts three types of architectural interventions: layer ablation, layer reordering, and parallel layer execution. Contribution/Results: We discover that middle layers exhibit strong functional uniformity and order invariance—enabling safe skipping, arbitrary reordering, or concurrent execution—thereby challenging the conventional assumption of strict layer-order dependency. Across diverse downstream tasks, skipping or parallelizing middle layers reduces inference latency by up to 30% on average, with accuracy degradation under 1%. This study is the first to empirically characterize the functional heterogeneity spectrum across Transformer layers, providing an interpretable, evidence-based foundation for model lightweighting, architectural compression, and novel variant design.

This work challenges the prevailing hypothesis that emergent capabilities in large language models (LLMs) are solely determined by model scale, seeking instead to identify their fundamental drivers. Method: We conduct controlled experiments across Transformer models of varying sizes—trained with identical architecture, pretraining corpus, and tokenization—to isolate the effects of scale from those of optimization progress. Contribution/Results: We find that pretraining loss, rather than parameter count, is a more fundamental predictor of emergence: downstream task performance aligns closely across scales at equivalent loss values, and sharp, task-specific accuracy jumps occur when loss falls below empirically determined thresholds—well above chance. We formally characterize emergence as a loss-driven phenomenon and propose a loss-threshold criterion to replace conventional discontinuity-based detection. Empirical validation confirms a strong correspondence between loss thresholds and the onset of emergent behavior, establishing a new paradigm for modeling emergence mechanisms.

Why do large language models (LLMs) require tokenization, and why does character-level modeling lead to performance degradation in Transformers? Method: The authors construct a *k*-order Markov data source and rigorously analyze the cross-entropy of Transformers under character-level versus token-level modeling, grounding the analysis in information-theoretic modeling capacity. They establish a provable relationship between tokenization strategies and the accuracy of sequence probability estimation. Contribution/Results: Theoretically, without tokenization, Transformers collapse to modeling only unigram character distributions, failing to capture higher-order dependencies; with appropriate tokenization, learning single-step token predictions suffices to near-optimally model the source distribution. Empirically, tokenization significantly reduces cross-entropy on high-order Markov sources. This work provides the first rigorous information-theoretic and probabilistic justification that tokenization is a necessary condition for overcoming the fundamental limitations of character-level Transformer modeling.

This work investigates the dynamic evolution of token representations in pretrained Transformers, focusing on how positional encoding schemes—absolute versus rotary—affect their continuous-time dynamical behavior. We propose a modeling framework grounded in nonlinear dynamical systems theory, rigorously deriving necessary and sufficient conditions for token representations to converge to zero or diverge. Our analysis is the first to systematically demonstrate that rotary positional encoding suppresses pathological convergence and enhances representation separation. Both theoretical analysis and empirical experiments confirm that excessive token convergence degrades model expressivity. Guided by these insights, we design lightweight architectural enhancements—such as dynamic attention scaling—that effectively mitigate convergence issues and yield consistent performance gains across multiple benchmarks. This work establishes a novel dynamical-systems perspective for understanding Transformer internals and provides interpretable, theory-backed principles for architecture optimization.

This paper investigates whether a single-layer Transformer without positional encoding possesses the universal approximation property (UAP) for vocabulary-in-context learning (VICL). Theoretically, we prove that in the absence of positional encoding, the model cannot achieve VICL-UAP; however, introducing positional encodings satisfying specific spectral conditions—such as sinusoidal encoding—strictly restores UAP. Our analysis is grounded in function approximation theory, where we formally model and analyze VICL capability via mathematical characterization of representational capacity. This work establishes, for the first time, a necessary and sufficient framework linking the existence of positional encoding to VICL-UAP. The results demonstrate, from an approximation-theoretic perspective, that positional encoding is both *necessary and sufficient* for VICL-UAP—not merely a heuristic aid for sequence modeling, but a fundamental theoretical prerequisite for contextual generalization. This provides a novel paradigm for understanding the essential role of positional information in Transformers.

This study investigates the marginal benefits of a three-stage strategy—self-supervised pretraining, intermediate fine-tuning, and downstream task adaptation—for small-scale Vision Transformers (~5M parameters). Motivated by the observation that intermediate fine-tuning may degrade downstream performance due to task misalignment, we propose a systematic ablation framework to assess the impact of varying dataset and objective combinations across stages. Experiments reveal that targeted pretraining substantially improves small-model performance, whereas introducing semantically distant intermediate tasks yields no gain—and often harms performance while wasting compute. The core contribution is the empirical demonstration that, for small ViTs, **the quality of data selection is far more critical than the number of stacked tasks**, challenging conventional assumptions about multi-stage transfer. This finding provides key empirical evidence and methodological guidance for designing efficient, lightweight self-supervised learning paradigms.

Standard supervised training often struggles to learn effective query-key attention patterns in Transformer-based sequence classification tasks, particularly failing to induce a preference for neighboring positions. This work demonstrates through systematic ablation studies and simplified theoretical analysis that self-pretraining (SPT), driven by a masked reconstruction objective, enables the model to acquire such localized attention structures from random initialization, substantially improving optimization dynamics. Without relying on external data, SPT significantly outperforms purely supervised training on benchmarks such as the Long-Range Arena. The performance gains are primarily attributed to the model’s enhanced ability to learn interactions among nearby tokens, highlighting proximity-aware attention as a key mechanism underlying SPT’s effectiveness.

High experimental costs and difficulties in conducting controlled, multi-condition studies hinder pretraining research for large language models (LLMs). To address this, we propose a “single-training, multiple-experiments” paradigm: ten heterogeneous experiments—including knowledge acquisition, mathematical reasoning, and others—are executed in parallel during a single 1.5B-parameter LLM pretraining run. Leveraging controlled-variable design, dynamic data injection, interactive detection, and contamination analysis, we ensure negligible cross-experiment interference. This approach dramatically improves research efficiency—reproducing established findings and enabling novel explorations—while incurring virtually no additional computational overhead or performance degradation, achieving up to 90% compute savings. Our core contribution is the first systematic realization of a scientific experimentation framework for LLM pretraining that supports concurrent multi-task learning, multi-hypothesis testing, and full reproducibility.