Task Diversity Shortens the ICL Plateau

2024-10-07

arXiv.org

AI Summary

Multi-task in-context learning (ICL) training often exhibits prolonged loss plateaus, contradicting the intuition that task stacking inherently increases optimization difficulty. Method: We construct controllable synthetic ICL task sets, perform multi-task joint training, and complement empirical analysis with theoretical studies on simplified neural networks and dynamic loss landscape tracking. Contribution/Results: We demonstrate that task diversity significantly improves training dynamics: across multiple synthetic tasks, average loss plateau duration shortens by over 40%, convergence steps decrease by more than 30%, and generalization performance consistently improves. This work provides the first evidence that intrinsic task diversity in natural language data is a key enabler of ICL success in large language models (LLMs). Our findings offer a novel explanation for efficient LLM training—diversity fosters smoother, faster optimization—thereby advancing the theoretical understanding of ICL and informing scalable model training strategies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In-context learning (ICL) describes a language model's ability to generate outputs based on a set of input demonstrations and a subsequent query. To understand this remarkable capability, researchers have studied simplified, stylized models. These studies have consistently observed long loss plateaus, during which models exhibit minimal improvement, followed by a sudden, rapid surge of learning. In this work, we reveal that training on multiple diverse ICL tasks simultaneously shortens the loss plateaus, making each task easier to learn. This finding is surprising as it contradicts the natural intuition that the combined complexity of multiple ICL tasks would lengthen the learning process, not shorten it. Our result suggests that the recent success in large-scale training of language models may be attributed not only to the richness of the data at scale but also to the easier optimization (training) induced by the diversity of natural language training data.

Problem

• Reduces ICL loss plateaus
• Enhances multi-task learning efficiency
• Contradicts complexity intuition in training

Innovation

• Diverse ICL tasks training
• Shortens loss plateaus
• Optimizes language model learning