SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

2026-08-04

AI Summary

This study addresses the puzzling observation that while supervised fine-tuning (SFT) in multi-task large language model training suffers from task conflicts due to gradient interference, reinforcement learning (RL) enables stable coexistence of tasks through an unclear mechanism. The work reveals, for the first time, that SFT induces significant gradient conflicts under parameter norm constraints, whereas RL mitigates this issue by leveraging advantage normalization and on-policy updates to effectively bound gradient variance, thereby aligning task-specific update directions nearly orthogonally. Building on this insight, the authors propose Parallel-RL, a novel training paradigm that decouples multi-task learning. Empirical results validate the sparsity and orthogonality of RL updates and demonstrate that Parallel-RL substantially improves both training efficiency and performance across multiple tasks.

Abstract

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.

Problem

• multi-task learning
• supervised fine-tuning
• reinforcement learning
• task conflict
• gradient interference

Innovation

• Reinforcement Learning
• Supervised Fine-Tuning
• Multi-Task Learning
• Gradient Interference
• Parallel-RL