Quantifying Behavioral Tails in Black-Box Language Models

2026-09-27

AI Summary

This study addresses the challenge of estimating extreme behavior probabilities in black-box large language models, which lack a tractable input distribution definition. To this end, we propose RareTrap, a framework that leverages surrogate models to construct a geometry-aware mapping from a low-dimensional latent space to the embedding space, thereby inducing an explicit prompt distribution. By integrating response-level performance functions with sequential rare-event simulation, our approach enables efficient tail risk assessment. Remarkably, RareTrap requires only 200 evaluations to quantify the probabilities of severe extreme behaviors—such as excessive resource consumption—across multiple models. This work provides a principled methodology for advancing the safety alignment of large language models.

Abstract

We introduce RareTrap, a framework for estimating the probability of severe behaviors in black box large language models (LLMs). A key challenge for probability estimation is defining a tractable distribution over the input space. To accomplish that, RareTrap uses a surrogate LLM and constructs a geometry-aware mapping from a lower-dimensional latent reference space into its token-embedding space to induce an explicit and reproducible distribution over input prompts. A response-level performance function is utilized on the response to quantify behavior severity. This enables sequential rare event simulation that concentrates evaluations on progressively more severe behaviors while preserving probability under the induced prompt distribution, which would otherwise be prohibitive to measure. Across 10 open-weight and two frontier models (GPT-5.4 and Claude Sonnet 4.6), we find that RareTrap successfully induces severe resource consumption behaviors and computes their probability with as few as 200 evaluations. RareTrap provides model developers a principled approach for evaluating language models under a common distribution, and prioritizing alignment effort to improve safety and mitigate risks. Code is published online: https://anonymous.4open.science/r/rare_trap-D7F0.

Problem

• black-box LLMs
• behavioral tails
• rare event estimation
• probability quantification
• AI safety

Innovation

• Rare event simulation
• Black-box LLMs
• Behavioral tails
• Geometry-aware mapping
• Probability estimation