Exploring Explanations Improves the Robustness of In-Context Learning

2025-06-03

AI Summary

Existing in-context learning (ICL) approaches for large language models exhibit weak out-of-distribution (OOD) generalization and are highly sensitive to distributional shifts in demonstration examples, leading to unreliable predictions. To address this, we propose X²-ICL—a novel ICL framework that, for the first time, extends explanation generation across the full candidate label space. It systematically produces reasoning paths for all possible labels and introduces a multi-label consistency evaluation mechanism to enable robust, post-hoc verification of decision processes. Crucially, X²-ICL requires no additional training or fine-tuning and operates purely via prefix-based inference. Extensive experiments on multiple natural language understanding benchmarks demonstrate that X²-ICL achieves an average 12.7% improvement in OOD accuracy over state-of-the-art ICL methods, significantly enhancing robustness against both distributional shifts and noisy demonstrations. This work establishes a new paradigm for interpretable and trustworthy in-context learning.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

In-context learning (ICL) has emerged as a successful paradigm for leveraging large language models (LLMs). However, it often struggles to generalize beyond the distribution of the provided demonstrations. A recent advancement in enhancing robustness is ICL with explanations (X-ICL), which improves prediction reliability by guiding LLMs to understand and articulate the reasoning behind correct labels. Building on this approach, we introduce an advanced framework that extends X-ICL by systematically exploring explanations for all possible labels (X$^2$-ICL), thereby enabling more comprehensive and robust decision-making. Experimental results on multiple natural language understanding datasets validate the effectiveness of X$^2$-ICL, demonstrating significantly improved robustness to out-of-distribution data compared to the existing ICL approaches.

Problem

• Enhancing robustness of in-context learning (ICL) with explanations
• Improving generalization beyond provided demonstration distributions
• Systematically exploring explanations for all possible labels (X$^2$-ICL)

Innovation

• Extends X-ICL with systematic explanation exploration
• Guides LLMs to articulate reasoning for all labels
• Improves robustness to out-of-distribution data