few-shot learning

This work systematically evaluates the generalization capability and representation robustness of small language models (SLMs) under two adaptation paradigms—few-shot prompting and supervised fine-tuning—focusing on low-resource settings, out-of-distribution (OOD) generalization, and multi-task scenarios. Methodologically, we integrate centered kernel alignment (CKA) and representational similarity analysis (RSA) for representation similarity quantification, complemented by OOD generalization benchmarks and multi-scale model comparisons. Our study is the first to characterize the knowledge internalization mechanisms of these paradigms through the lenses of representation stability and abstraction level. Results show that prompt-based learning yields more flexible representations but exhibits fragile OOD generalization; in contrast, fine-tuning achieves greater robustness yet suffers from overfitting and reduced abstraction depth. These findings provide interpretable theoretical foundations and empirically grounded guidelines for selecting adaptation strategies for SLMs in resource-constrained environments.

Instruction tuning often causes pretrained language models to forget foundational knowledge and over-specialize in conversational patterns, thereby degrading in-context learning (ICL) performance. This work identifies an intrinsic trade-off between instruction-following capability and ICL ability. To address it, we propose *partial adaptation*, a lightweight, parameter-efficient tuning paradigm: leveraging LoRA or Adapter modules, we freeze subsets of model parameters and progressively scale adaptation strength—without additional training or extra parameters. Evaluated across 12 canonical NLP few-shot tasks, our method improves average accuracy by 4.2%, while incurring only a marginal drop (−1.8%) in AlpacaEval instruction-following scores. Notably, this is the first systematic study to characterize performance trajectories across multiple model families and scales. Our approach offers a scalable, low-overhead pathway to balance instruction alignment with generalization—preserving ICL competence without compromising task-specific fidelity.

This work proposes a training-free inference method for few-shot test-time adaptation under the challenging setting where the model is entirely frozen and source data are inaccessible. The approach performs a measure transformation on the encoder’s latent embedding distribution via exponential tilting, leveraging the support set to compute task similarity scores and reweighting the distribution according to a KL-optimal criterion to adapt to the new task. As the first method to achieve training-free test-time adaptation with a completely frozen model, it relies solely on inference-stage distribution correction. Extensive experiments demonstrate that the proposed method matches or even surpasses the performance of approaches requiring parameter updates across multiple few-shot benchmarks, thereby validating the effectiveness and potential of purely inference-level adaptation.

Soft prompt tuning often suffers from catastrophic forgetting of general-purpose knowledge in vision-language models (e.g., CLIP) under few-shot settings, leading to performance worse than zero-shot inference. Method: We propose Gradient Alignment (GA), a novel optimization mechanism that constrains prompt gradient updates to align with the direction of zero-shot predictions derived from predefined prompts—thereby explicitly preserving task-agnostic, pre-trained knowledge without requiring additional data, regularization, or architectural modifications. Contribution/Results: GA effectively mitigates overfitting and inter-class interference. It consistently outperforms state-of-the-art prompt-tuning methods across diverse transfer scenarios—including few-shot learning, domain generalization, base-to-novel class adaptation, and cross-dataset transfer—delivering substantial improvements in both generalization stability and accuracy.

Large language models (LLMs) face significant challenges in task adaptation under resource-constrained and closed-source API settings, where conventional parameter-efficient fine-tuning (PEFT) methods are inapplicable due to their reliance on direct model parameter access and high computational overhead. Method: This paper proposes a lightweight, parameter-free knowledge injection framework that enables task-specific adaptation without accessing the LLM’s internal parameters. Its core innovation is the “Specialized Small Model (SSM) Collaboration Paradigm,” integrating knowledge distillation from the LLM, distribution-aware task modeling, and zero-parameter coupling between the SSM and the LLM. Contribution/Results: Experiments demonstrate that our approach matches PEFT-level performance across diverse downstream tasks while reducing GPU memory consumption by over 90% and inference latency by 85%. Crucially, it operates entirely within black-box API environments—requiring no model weights, gradients, or architectural access—thus enabling seamless integration with proprietary, closed-source LLM APIs.

This work addresses the challenge of suppressing undesirable behaviors—such as sudden alignment failures—learned during model training while preserving desired capabilities and avoiding unintended backdoors. The authors propose the Inoculation Adapter (IA) method, which first trains a LoRA adapter specialized in capturing undesirable behaviors, then freezes this adapter to guide the training of the main task adapter. Only the main adapter is deployed, thereby reducing the optimization pressure that leads the model to acquire undesirable capabilities. Unlike prompt-based inoculation, IA effectively mitigates behaviors that are difficult to elicit via prompting and substantially diminishes the risk of accidental backdoors. Experiments across six model families demonstrate that IA achieves more selective capability suppression while enhancing both safety and general applicability.

This work addresses the base-new class trade-off in few-shot fine-tuning of audio-language models, where zero-shot drift in the text embedding space often improves performance on seen classes at the expense of generalization to unseen ones. To mitigate this issue, the authors propose Subspace Tuning (SubT), a novel framework that introduces geometric constraints into few-shot adaptation for the first time. SubT employs structured subspace parameterization to limit embedding deformation and incorporates a residual anchoring mechanism to stabilize adaptation relative to the zero-shot prior. During inference, a subspace-aware gating strategy suppresses negative transfer for weakly aligned unseen classes. Without requiring backpropagation through the text encoder, SubT achieves significant improvements in few-shot generalization across eleven audio benchmarks while maintaining computational efficiency.

This study systematically investigates the effectiveness and limitations of test-time adaptation methods that do not require updating model parameters in open-source large language models. Focusing on many-shot in-context learning (ICL), it integrates dynamic and reinforcement-based ICL prompting strategies to evaluate how the number, ordering, and selection mechanisms of examples influence performance across diverse tasks and model architectures. The findings reveal that many-shot prompting substantially improves performance on structured tasks with high information gain but is highly sensitive to example selection, whereas its benefits are limited in open-ended generation tasks. This work delineates the applicability boundaries and potential risks of prompt-based test-time adaptation, offering both theoretical grounding and practical guidance for real-world deployment.

This work addresses the limitations of standard alignment fine-tuning, which often yields shallow alignment and poor generalization due to its reliance on demonstration data that inadequately captures the underlying norms governing desired behavior. To overcome this, the authors propose Model Spec Midtraining (MSM), a method that injects synthetic documents encoding explicit normative knowledge—defined by a Model Spec—between pretraining and alignment fine-tuning. This intermediate training stage enables the model to internalize structured norms and thereby generalize target behaviors more effectively from subsequent demonstrations. The study demonstrates that the formulation of norm statements, particularly those incorporating concreteness and value-based explanations, critically influences generalization performance. Evaluated on Qwen3-32B, MSM reduces agentive misalignment rates from 54% to 7%, substantially outperforming deliberative alignment baselines (14%).