Deep Multi-Task Learning Has Low Amortized Intrinsic Dimensionality

2025-01-31

AI Summary

This paper investigates efficient modeling and generalization mechanisms for deep multi-task learning under parameter constraints. Addressing the empirical observation that multi-task models reside in low-dimensional subspaces, we propose a low-dimensional parameterization method based on random expansion, substantially reducing the number of free parameters. We derive the first non-vacuous PAC-Bayesian generalization bound for deep multi-task networks, theoretically establishing that their amortized intrinsic dimension is lower than that of single-task counterparts. Through joint analysis of intrinsic dimension estimation, weight compression, and random expansion, we demonstrate that the method achieves significant parameter reduction—average intrinsic dimension markedly below total parameter count—while preserving high multi-task accuracy. Moreover, it provides the first theoretically grounded characterization of generalization performance for such compressed multi-task architectures.

Technology Category

• Machine Learning: Learning with Manifolds
• Data Mining & Knowledge Management: Data Compression
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Deep learning methods are known to generalize well from training to future data, even in an overparametrized regime, where they could easily overfit. One explanation for this phenomenon is that even when their *ambient dimensionality*, (i.e. the number of parameters) is large, the models' *intrinsic dimensionality* is small, i.e. their learning takes place in a small subspace of all possible weight configurations. In this work, we confirm this phenomenon in the setting of *deep multi-task learning*. We introduce a method to parametrize multi-task network directly in the low-dimensional space, facilitated by the use of *random expansions* techniques. We then show that high-accuracy multi-task solutions can be found with much smaller intrinsic dimensionality (fewer free parameters) than what single-task learning requires. Subsequently, we show that the low-dimensional representations in combination with *weight compression* and *PAC-Bayesian* reasoning lead to the first *non-vacuous generalization bounds* for deep multi-task networks.

Problem

• Deep Multi-task Learning
• Parameter Efficiency
• Generalization Ability

Innovation

• Multi-task Learning
• Parameter Efficiency
• PAC-Bayesian Inference