Estimating Rare Events in Language Models with Proper Evaluation

2026-07-20

AI Summary

This work addresses the challenge of estimating rare yet high-risk failures of language models under distribution shift, which conventional random sampling struggles to capture due to their low probability. The authors propose Gradient-Activated Adaptive Multilevel Splitting (GA-AMLS), a method that operates in a continuous activation space by integrating a gradient-guided MCMC kernel with a heavy-tailed prior. This approach circumvents the zero-estimation collapse often caused by input-space search and dispenses with the independence assumptions inherent in existing estimators. Additionally, they introduce the Shifted-Power Bregman (SPB) loss function, which enables asymmetric error penalization and proper scoring even under zero-estimate conditions. Experiments demonstrate that GA-AMLS substantially reduces mean squared error in log-space on small-scale Transformers, highlighting the necessity of aligning estimation performance with the cost structure of deployment scenarios.

Abstract

Quantifying the risk of rare failures in language models, such as those triggered by adversarial distribution shifts or very large-scale deployments, requires estimating probabilities far too small for random sampling. While recent work has formalized Low Probability Estimation, existing pipelines remain fragile in the rarest regimes: estimators can suffer zero-estimate collapse or systematic bias, and standard evaluation losses can become unstable or poorly matched to asymmetric safety costs. In this work, we introduce Gradient Activation Adaptive Multi-Level Splitting (GA-AMLS), which adapts rare-event Monte Carlo methods to the continuous activation space of language models. Specifically, GA-AMLS uses a gradient-based MCMC kernel to navigate activation space, eliminating the zero-estimate collapse of input-space search and replacing the independence assumptions of prior activation-space estimators with conditional sampling under an explicit, heavier-tailed activation prior. We also propose the Shifted-Power Bregman (SPB) Loss, a proper scoring rule that remains finite for zero-estimates and offers tunable asymmetry between underestimation and overestimation penalties. Experiments on small transformer models reveal a bias-variance tradeoff: GA-AMLS achieves the lowest loss under symmetric evaluation, reducing average log-space squared error relative to the strongest baseline across model sizes, while methods with overestimation bias prevail under asymmetric penalties. Our findings highlight that estimator choice should be matched to deployment context. More broadly, our work establishes activation space as a tractable domain for rare-event estimation in language models, circumventing the brittleness of discrete input-space search.

Problem

• Rare Event Estimation
• Language Models
• Low Probability Estimation
• Adversarial Distribution Shift
• Proper Scoring Rules

Innovation

• rare-event estimation
• activation space
• gradient-based MCMC
• proper scoring rule
• asymmetric loss