StabilityBench: Benchmarking Instability in LLMs

2026-07-17

AI Summary

This work addresses the limitations of current large language model (LLM) evaluations, which predominantly rely on static, single-turn benchmarks and fail to capture performance instability arising from contextual shifts in multi-turn dialogues. To this end, we propose StabilityBench—a general, model-agnostic benchmarking framework that transforms single-turn tasks into dynamic multi-turn interactions by incorporating user simulators, such as demographic personas or flattery-based decoys, thereby injecting context perturbations while preserving the original task intent. We introduce, for the first time, a systematic mechanism for multi-turn context perturbation and accompany it with a lightweight variant, StabilityBench-Mini, to balance evaluation efficiency and diversity. Experiments across nine mainstream LLMs reveal significant performance degradation in three-quarters of existing benchmarks, underscoring the inadequacy of static evaluation and demonstrating that StabilityBench effectively uncovers LLM instability under dynamic interaction scenarios.

Abstract

AI Assistants are increasingly deployed in high-stakes settings, such as healthcare or government services. Yet their real-world behavior remains poorly understood due to strong context dependence. Current evaluation protocols follow a defense-in-depth paradigm with compounding layers of safeguards, ranging from traditional benchmarks to live or adversarial testing. Such benchmarks remain largely static and single-turn, limiting their ability to capture real-world variability in conversational settings. We propose StabilityBench, a principled, general and model-agnostic benchmark operator that transforms single-turn benchmark queries into multi-turn interaction histories. StabilityBench augments existing benchmarks by injecting realistic user simulations, through demographic proxies or sycophantic baits, while preserving original task intent. We apply StabilityBench to four benchmarks spanning mathematical reasoning, health question-answering and safety, and evaluate nine large language models under these conditions. Our results show that model performance is consistently unstable under these injections, with considerable performance degradations on three out of four benchmarks studied. These highlight important limitations of static evaluations and motivate more realistic evaluation settings. To this end, we propose StabilityBench-Mini: a size-preserving variant of StabilityBench that samples across diversification axes, enabling more realistic evaluation without increasing costs.

Problem

• LLM instability
• benchmarking
• context dependence
• multi-turn interaction
• real-world evaluation

Innovation

• StabilityBench
• multi-turn interaction
• model-agnostic benchmarking
• realistic user simulation
• evaluation instability