Stress Testing Concept Erasure with Large Language Model Agents

2026-07-20

AI Summary

Existing evaluation methods for concept erasure rely on static, handcrafted probes that are biased and lack scalability, failing to comprehensively uncover failure modes of models under diverse natural language inputs. To address this limitation, this work proposes STACE, a novel framework that introduces, for the first time, an adaptive hypothesis search mechanism into concept erasure evaluation. STACE establishes an automated stress-testing paradigm based on multi-agent large language models, wherein multiple agents collaboratively and iteratively generate, critique, and validate hypotheses enriched with external knowledge to systematically probe vulnerabilities in erased models. Experiments demonstrate that STACE significantly outperforms five baseline methods across four concept categories, encompassing two text-to-image models, six erasure techniques, and multiple intensity settings. The framework exhibits exceptional robustness, generalization, and scalability, and can be readily extended to tasks such as LLM jailbreaking.

Abstract

Concept erasure aims to remove semantic concepts from a trained generative model and is increasingly important for responsible AI deployment. However, verifying whether a model has robustly removed targeted concepts remains a critical challenge. Existing evaluation methods are typically pre-defined and static, failing to expose vulnerabilities under diverse natural-language probes and challenging conditions. Moreover, manually designed evaluation strategies can be biased and difficult to scale. We posit that concept erasure evaluation is best formulated as an adaptive hypothesis search, operationalised by agents that iteratively propose, critique, and verify tests to systematically expand coverage of failure modes. To this end, we propose Stress Testing Agents for Concept Erasure (STACE), a framework that autonomously stress-tests concept-erased models using multiple Large Language Model (LLM) agents, by iteratively generating and verifying stress-testing hypotheses grounded by external knowledge. We also introduce a suite of metrics for assessing the performance and efficiency of LLM-agent-powered stress-testing frameworks. Our extensive experiments show that STACE outperforms five LLM-based evaluation baselines on four concept categories. Further analysis across two T2I models, six concept erasure approaches, and various erasure strengths show that STACE is robust for different settings. We also show that STACE can be adapted beyond concept erasure evaluation to other problem domains, such as LLM jailbreaking. Our code is available anonymously.

Problem

• concept erasure
• stress testing
• large language models
• evaluation robustness
• failure mode detection

Innovation

• concept erasure
• stress testing
• LLM agents
• adaptive hypothesis search
• responsible AI