When Behavioral Safety Evaluation Fails: A Representation-Level Perspective

2026-06-06

AI Summary

Current behavioral safety evaluations struggle to capture the representational vulnerabilities of large language models, resulting in critical assessment blind spots. This work proposes a soft intervention–based evaluation framework that applies controlled perturbations—such as harmful fine-tuning and layer-wise latent perturbations—in both parameter and latent spaces. Introducing the concept of “audit gap” and a Latent Vulnerability Score (LVS), the study systematically reveals, for the first time, a significant inconsistency between behavioral safety and representational robustness. Experimental results demonstrate that even when model behavior appears normal, the LVS markedly increases, with intermediate-layer representations exhibiting the highest sensitivity to interventions. These findings underscore that reliance solely on behavioral metrics is insufficient for a comprehensive safety assessment of language models.

Abstract

Large Language Model (LLM) safety has often been evaluated at the behavior level, which provides limited evidence of internal robustness, as these evaluations target outputs rather than representation-level vulnerability under intervention. We formalize this discrepancy as the audit gap: the difference between behavioral safety and robustness under intervention. To study this gap, we construct dissociated models that preserve safe outward behavior while remaining vulnerable in the latent space. We introduce an intervention-based evaluation framework to test model robustness through soft interventions in parameter and latent spaces, including harmful fine-tuning and layer-wise latent perturbations. To formalize the evaluation, we propose the Latent Vulnerability Score (LVS) to measure how easily harmful behavior can be elicited by bounded latent perturbations. Using this evaluation framework, we show that behavioral safety metrics are insufficient measures of representation-level robustness across multiple safely and unsafely aligned state-of-the-art models. Notably, dissociated models show substantially elevated LVSs despite comparable refusal behavior under harmful intervention, with intermediate representations being the most sensitive to intervention. Our results suggest that behavioral safety evaluation alone provides an incomplete picture of model robustness, motivating representation-aware audits of latent vulnerability and observable behavior.

Problem

• Behavioral Safety
• Representation-Level Robustness
• Audit Gap
• Latent Vulnerability
• Intervention

Innovation

• audit gap
• latent vulnerability
• intervention-based evaluation
• dissociated models
• Latent Vulnerability Score