Learning Perturbations to Extrapolate Your LLM

2026-05-13

AI Summary

This work addresses the limited out-of-distribution extrapolation capability of current large language models, which stems from their reliance on fixed discrete perturbations that lack flexibility. To overcome this limitation, the authors propose a continuous, learnable perturbation framework that introduces trainable latent vectors in the embedding space to flexibly perturb token prefixes. The method integrates unbiased estimating equations with stochastic gradient descent for optimization and provides a theoretical analysis of the estimator’s statistical properties under over-parameterization theory. Experimental results demonstrate that the proposed approach significantly outperforms multiple state-of-the-art baselines on both synthetic and real-world datasets, achieving robust performance gains in out-of-distribution settings.

Abstract

Recent advancements in large language models demonstrate that injecting perturbations can substantially enhance extrapolation performance. However, current approaches often rely on discrete perturbations with fixed designs, which limits their flexibility. In this work, we propose a framework where token prefixes are perturbed by a learnable transformation of a continuous latent vector within an embedding space. To overcome the challenge of an intractable marginal likelihood, we derive unbiased estimating equations for model parameters and optimize them via stochastic gradient descent. We establish the statistical properties of the resulting estimator in over-parameterized regimes. Empirical evaluations on both synthetic and real-world datasets demonstrate that our proposal yields significant gains in out-of-domain settings over a range of state-of-the-art baseline methods.

Problem

• extrapolation
• large language models
• perturbations
• out-of-domain generalization
• flexibility

Innovation

• learnable perturbations
• continuous latent vector
• extrapolation
• large language models
• unbiased estimating equations