🤖 AI Summary
General-purpose large language models (LLMs) exhibit deficiencies in domain-specific knowledge (e.g., finance), mathematical reasoning, and multilingual capabilities. Method: We propose constructing a high-performance financial-domain LLM by fusing multiple domain-specific continual pretraining (CPT) expert models—avoiding costly and unstable end-to-end multi-skill training. Contribution/Results: This work presents the first systematic study on CPT model fusion, introducing a three-stage evaluation framework (knowledge recovery, skill complementarity, capability emergence) and benchmarking Task Arithmetic, TIES, and DARE-TIES on an 18-task financial evaluation suite. Fusion effectively restores general knowledge, improves overall performance, and induces emergent cross-domain capabilities. TIES demonstrates superior robustness, while Task Arithmetic achieves strong performance but is highly sensitive to hyperparameters. Our framework establishes principled, efficient pathways for building multi-competency domain LLMs from existing expert model assets.
📝 Abstract
While LLMs excel at general tasks, they struggle in specialized domains like finance, requiring diverse skills in domain knowledge, mathematical reasoning, and multilingual processing. Merging domain-specific Continual Pre-training (CPT)"experts"offers a practical alternative to costly and unstable multi-skill training. However, unlike established Supervised Fine-Tuning (SFT) model-based merging, CPT model merging remains largely unexplored. We address this gap by creating financial LLMs from experts in finance, math, and Japanese. We propose a three-stage evaluation focusing on knowledge recovery, complementarity, and emergence, and assess three merging methods (Task Arithmetic, TIES, and DARE-TIES) on a comprehensive financial benchmark curated from 18 tasks across 8 established datasets. Results show that merging an expert with its base model recovers general knowledge lost during CPT, while merging experts improves performance and can yield emergent cross-domain skills. Among the methods, Task Arithmetic performs strongly but is hyperparameter-sensitive, whereas TIES is more robust. Our findings also suggest that while model similarity correlates with merging success, emergent skills depend on more complex factors. This work presents the first foundational analysis of CPT model merging, establishing a principled framework and providing clear guidance for building multi-skill LLMs from existing assets.