continued pretraining

This work addresses the performance degradation of existing language models on certain tasks and the high computational cost of mainstream adaptation methods that rely on training. The authors propose a novel paradigm termed “model transplantation,” which identifies task-relevant local functional modules within a source model through activation analysis and directly transfers them to a target model—enabling performance improvement without any training. This approach provides the first empirical evidence that language models possess internally localized task-specific structures, facilitating cross-model capability transfer. Experiments demonstrate that transplanting modules between models across generations or between base and instruction-tuned variants can achieve up to 2.33× the baseline performance, with full recovery (100%) of task performance gaps in some cases.

Adapting small language models (125M parameters) to the education domain under constrained computational resources remains challenging due to high training costs and catastrophic forgetting during fine-tuning. Method: This paper proposes a continual pre-training paradigm tailored for small models, leveraging 0.4–1.0 billion tokens of domain-specific educational text. It integrates data cleaning, gradient checkpointing, and mixed-precision training to optimize memory usage while preserving general linguistic capabilities and enhancing domain specialization. Contribution/Results: The approach achieves superior token efficiency and domain adaptation without full retraining. On multi-benchmark evaluations—MMLU and HellaSwag—it improves accuracy by 8.1% and 7.6%, respectively, outperforming standard baselines. Empirical results demonstrate that lightweight continual pre-training is a viable, computationally efficient alternative for domain adaptation of small models in resource-constrained settings, offering a practical pathway for building domain-specific language models in education.

This study addresses the challenge of continual pretraining of large language models (LLMs) in dynamic knowledge environments, aiming to balance assimilation of new knowledge with retention of prior knowledge. To this end, we introduce the first benchmark specifically designed for evaluating continual pretraining under evolving data distributions, enabling systematic analysis of the interplay among model scale, semantic structure of domain sequences, and knowledge transfer/forgetting. We propose a novel cross-domain adaptive evaluation paradigm and uncover three key findings: (i) smaller models (<1.5B parameters) exhibit high sensitivity to both learning and forgetting; (ii) semantically ordered domain sequences foster specialization, whereas random sequences enhance generalization and cross-domain transfer; and (iii) larger models consistently achieve lower perplexity. Empirical results demonstrate that our continual pretraining paradigm significantly improves downstream task performance across the GPT-2 family, with particularly pronounced gains for smaller models.

To address the dual challenges of “new-language enhancement” and “original-language capability preservation” in cross-lingual adaptation of self-supervised speech models, this paper proposes a LoRA-driven framework for language-incremental expansion. Methodologically, it integrates Low-Rank Adaptation (LoRA) with a dual-track capability retention strategy: (i) multilingual data mixing during fine-tuning and (ii) k-means re-clustering to optimize the discrete representation space. The approach is instantiated on the mHuBERT architecture to enable efficient Chinese extension. Experiments demonstrate that only 0.3% of mHuBERT’s parameters require tuning for Chinese integration, yielding a MOS improvement of 1.6 and a relative WER reduction of 61.72%, while preserving zero performance degradation across all pre-existing language tasks. To our knowledge, this is the first work to systematically introduce LoRA into progressive multilingual expansion of self-supervised speech models, achieving a favorable trade-off among parameter efficiency, cross-lingual compatibility, and capability stability.

Current multi-stage training paradigms for large language models (LLMs) hinder causal attribution of design choices across stages. To address this, we introduce EvoLM—a comprehensive model suite systematically dissecting the four canonical training phases: pretraining, continued pretraining, supervised fine-tuning, and reinforcement learning—with over 100 models spanning 1B and 4B parameter scales. EvoLM enables the first fully transparent, reproducible analysis of end-to-end training dynamics, supported by a unified training-evaluation pipeline for ablation studies, large-scale from-scratch training, multi-stage evaluation, and in-domain/out-of-domain generalization assessment. We publicly release all models, stage-specific datasets, and code. Key findings include: (1) continued pretraining serves as a critical bridge enabling performance leaps; (2) excessive training exhibits pronounced diminishing returns; and (3) catastrophic forgetting is effectively mitigated via inter-stage data reweighting. These results provide interpretable, reproducible guidance for downstream practitioners in configuring optimal multi-stage training pipelines.

This work addresses the challenge of efficiently extending large language models to new languages, which typically demands substantial pretraining and alignment data—often prohibitively expensive—and where existing data-free merging approaches struggle to balance retention of original capabilities with acquisition of new linguistic competence. The authors propose PARAMΔ integration, a method that upgrades a dense model to a mixture-of-experts (MoE) architecture by assigning language-specific experts and grafting post-training parameter deltas (Δ_post) to enable effective language expansion without additional alignment data. Under identical computational or parameter budgets, this approach significantly outperforms baseline methods, simultaneously preserving performance in original languages and enhancing capabilities in newly added ones. The technique demonstrates broad applicability across diverse models and post-training scenarios, overcoming the inherent trade-offs that limit conventional fusion strategies.

This work addresses the high cost, poor scalability, and catastrophic forgetting associated with conventional monolithic post-training when extending large language models to new domains. The authors propose BAR, a modular post-training framework in which domain-specific experts—trained independently through intermediate training, supervised fine-tuning, and reinforcement learning—are integrated into a Mixture-of-Experts architecture via a lightweight routing mechanism. BAR enables independent expert updates, linearly scalable training costs, and mitigation of performance degradation across tasks. Evaluated on a 7B-parameter model spanning mathematics, code generation, tool usage, and safety, the approach achieves a composite score of 49.1, matching or surpassing traditional full retraining baselines while substantially reducing update complexity and computational overhead.

This work addresses the prevalent issue of capability degradation and catastrophic forgetting in large language models following task-specific fine-tuning. To mitigate this, the authors propose Activation-difference-guided Channel Targeting (ACT), a method that identifies a sparse, decoupled, and stable subset of model channels where task-specific capabilities are concentrated. By selectively transferring only these critical channel parameters, the approach enables efficient capability fusion and recovery of forgotten skills. Experimental results demonstrate that ACT effectively preserves original competencies while restoring lost abilities across multilingual mathematical and scientific reasoning tasks, and successfully consolidates multiple specialized models into a single, versatile model without significant performance trade-offs.

This work addresses the limited instruction-following capability of large language models in low-resource languages and the high computational cost and data demands of conventional adaptation methods. It presents the first systematic investigation into the effectiveness of model weight merging for cross-lingual instruction transfer, directly fusing a general-purpose instruction-tuned model with a target-language base model—without requiring language-specific instruction data or repeated fine-tuning. Evaluated across four Iberian languages and two dominant architectures, the approach achieves instruction-following performance on par with traditional methods while substantially reducing computational overhead, thereby demonstrating the efficiency, generality, and scalability of weight merging as a viable strategy for low-resource multilingual adaptation.

This study addresses the high cost and opaque mechanisms associated with adapting large language models to new languages, as well as the limited understanding of how linguistic competence emerges during training. By analyzing the evolution of input comprehension and output generation capabilities in decoder-only Transformers during low-resource language fine-tuning, this work reveals—for the first time—a layer-wise specialization pattern wherein language perception and production functions become distinctly localized across model layers. Building on this insight, the authors propose CogSym, a heuristic strategy that fine-tunes only the top and bottom 25% of layers. This approach consistently achieves 97–98% of full-model fine-tuning performance across multiple adaptation methods, including LoRA and full-parameter fine-tuning, substantially reducing the computational cost of multilingual adaptation.