CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representations

2026-06-01

AI Summary

This work addresses the challenge in multi-task learning where shared representations often entangle task-relevant structures with task-irrelevant variations, leading to negative transfer and degraded generalization. To mitigate this, the paper introduces the first causality-driven representation disentanglement framework, which decomposes features into orthogonal semantic and residual streams via semantic-residual orthogonal decomposition. Task-relevant semantics are concentrated in the semantic stream, while irrelevant variations are isolated in the residual stream. By integrating physical priors with statistical constraints, the method effectively suppresses inter-task interference without requiring explicit gradient manipulation and yields a tighter bound on out-of-distribution generalization. Extensive experiments across multiple visual multi-task benchmarks demonstrate consistent and significant improvements over existing approaches under both in-distribution and out-of-distribution settings.

Abstract

Multi-task learning (MTL) aims to construct a joint model for multiple tasks by sharing a common representation across domains. To achieve this goal, existing optimization-centric methods either balance task gradients or modify the shared architecture. However, as these approaches remain agnostic to the content of the shared representation, they fail to disentangle task-relevant structure from spurious context, leading to negative transfer and poor generalization. To overcome this limitation, we propose Causal Orthogonal Representations for Multi-Task Learning (CORE-MTL), a causally motivated representation-centric framework that encourages a structured semantic-residual factorization of the shared representation, concentrating task-relevant structure in the semantic stream while relegating nuisance variation to the residual stream. We instantiate this framework in the visual domain by leveraging physical priors for structured scenes and statistical constraints for attributes. Theoretically, our method enjoys a tighter out-of-distribution generalization bound than optimization-centric methods and reduces task gradient interference without explicit gradient projection or reweighting. Empirically, CORE-MTL consistently outperforms existing methods on visual multi-task benchmarks in both in-distribution and out-of-distribution settings. Code is publicly available at https://github.com/Hope-Rita/CORE-MTL.

Problem

• multi-task learning
• shared representation
• negative transfer
• generalization
• spurious correlation

Innovation

• Causal Orthogonal Representations
• Semantic-Residual Factorization
• Multi-Task Learning
• Out-of-Distribution Generalization
• Gradient Interference Reduction