Evaluating Generalization and Representation Stability in Small LMs via Prompting

2025-06-15

AI Summary

This work systematically evaluates the generalization capability and representation robustness of small language models (SLMs) under two adaptation paradigms—few-shot prompting and supervised fine-tuning—focusing on low-resource settings, out-of-distribution (OOD) generalization, and multi-task scenarios. Methodologically, we integrate centered kernel alignment (CKA) and representational similarity analysis (RSA) for representation similarity quantification, complemented by OOD generalization benchmarks and multi-scale model comparisons. Our study is the first to characterize the knowledge internalization mechanisms of these paradigms through the lenses of representation stability and abstraction level. Results show that prompt-based learning yields more flexible representations but exhibits fragile OOD generalization; in contrast, fine-tuning achieves greater robustness yet suffers from overfitting and reduced abstraction depth. These findings provide interpretable theoretical foundations and empirically grounded guidelines for selecting adaptation strategies for SLMs in resource-constrained environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We investigate the generalization capabilities of small language models under two popular adaptation paradigms: few-shot prompting and supervised fine-tuning. While prompting is often favored for its parameter efficiency and flexibility, it remains unclear how robust this approach is in low-resource settings and under distributional shifts. This paper presents a comparative study of prompting and fine-tuning across task formats, prompt styles, and model scales, with a focus on their behavior in both in-distribution and out-of-distribution (OOD) settings. Beyond accuracy, we analyze the internal representations learned by each approach to assess the stability and abstraction of task-specific features. Our findings highlight critical differences in how small models internalize and generalize knowledge under different adaptation strategies. This work offers practical guidance for model selection in low-data regimes and contributes empirical insight into the ongoing debate over prompting versus fine-tuning. Code for the experiments is available at the following

Problem

• Assess generalization of small LMs via prompting and fine-tuning
• Compare robustness in low-resource and OOD settings
• Analyze representation stability across adaptation strategies

Innovation

• Compares few-shot prompting and fine-tuning techniques
• Analyzes generalization in low-resource and OOD settings
• Evaluates representation stability across adaptation strategies