Efficient Safety Benchmarking via Item Response Theory

2026-05-26

arXiv.org

AI Summary

This study addresses the high evaluation costs of traditional large language model (LLM) safety benchmarks, which suffer from item redundancy and limited discriminative power. By introducing Item Response Theory (IRT), this work uncovers interpretable latent structures within safety benchmarks and proposes two cost-efficient evaluation paradigms: dynamic adaptive item selection and static fixed subset extraction. Experimental results demonstrate that these approaches reduce evaluation costs by 80% to 99.9% while maintaining model ranking correlations exceeding 0.90. Furthermore, the proposed method effectively differentiates safety capabilities among top-performing models. Overall, this research presents a psychometric framework that achieves both efficiency and precision for LLM safety evaluation.

Abstract

Safety benchmarks for language models are typically evaluated using static paradigms that treat all items as equally informative for all models, an assumption that is particularly problematic for adversarial, highly heterogeneous safety items. Applied in full to modern benchmark suites, the current evaluation procedures would require on the order of $10^5$ responses, most of which provide little ranking signal. We analyze a suite of widely used safety benchmarks and make three contributions toward more efficient safety evaluation. First, we show that Item Response Theory (IRT) recovers interpretable structure on safety benchmarks, with ability estimates resolving differences among models that cluster at the ceiling of raw safety metrics. Second, we show that adaptive item selection, which dynamically chooses informative items for each model based on its responses, approximates full-benchmark rankings while reducing evaluation cost by at least 80% on benchmarks where Spearman's $\rho>$90% with full-benchmark is attainable, and by up to 99.9% on AIR-Bench 2024. Third, we introduce a practical procedure for extracting a fixed, informative subset of items reusable across models, providing an alternative to adaptive selection with savings of up to 99.8% on AIR-Bench 2024. Together, these results establish that psychometric methods enable benchmark-aware reductions in evaluation costs across the safety evaluation pipeline.

Problem

• safety benchmarking
• language models
• evaluation efficiency
• static evaluation paradigm

Innovation

• Item Response Theory
• Safety Benchmarking
• Adaptive Item Selection
• Evaluation Efficiency
• Psychometrics