Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation

2026-07-16

AI Summary

This work addresses the challenge of effectively training reasoning language models in domains lacking reliable verifiers and underutilizing abundant supervision data devoid of reasoning traces. The authors propose a low-cost, efficient approach that combines instruction fine-tuning with model merging to inject supervisory signals without requiring explicit reasoning trajectories, while preserving the model’s original reasoning capabilities through merging. This method achieves, for the first time, cross-domain transfer of reasoning abilities without performance degradation, yielding significant improvements on tasks such as code generation and text summarization at a cost under \$3, all while maintaining strong reasoning performance in other domains.

Abstract

Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement learning that drives RLM performance gains. However, training RLMs on domains that lack reliable verifiers remains challenging. Meanwhile, for both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist. In this work, we show that these data can be used efficiently to further improve RLM performance. For this, we first use classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM. Next, we merge our instruction-tuned model with the original reasoning model, recovering its reasoning behavior on the target domain. Our extensive evaluation demonstrates that our technique improves RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains. Importantly, our method is highly cost-effective, enabling such improvements for less than USD $3.

Problem

• reasoning language models
• instruction tuning
• supervised fine-tuning
• verifiable domains
• unverifiable domains

Innovation

• instruction tuning
• model merging
• reasoning language models
• supervised fine-tuning
• cost-effective adaptation