content safety

The field of safety evaluation for large language models (LLMs) lacks a systematic, unified survey. Method: This paper introduces the first holistic “Why/What/Where/How” four-dimensional analytical framework to rigorously distinguish safety evaluation from general model evaluation. Through bibliometric analysis, cross-methodological comparison, and taxonomy-driven modeling, it systematically synthesizes over 100 studies spanning core safety dimensions—including toxicity, bias, robustness, and truthfulness—and integrates diverse paradigms such as human evaluation, LLM-as-a-judge automation, red-teaming, and adversarial prompt engineering. Contributions: (1) A novel multi-granularity knowledge graph of LLM safety evaluation; (2) A structured resource inventory comprising 30+ benchmarks, 50+ metrics, and 20+ tools; and (3) Identification of key open challenges alongside reusable methodological pathways—providing both theoretical foundations and practical guidance for academic research and industrial deployment.

Existing text-to-image (T2I) models are vulnerable to misuse for generating unsafe content, while mainstream safety mechanisms exhibit poor robustness under distribution shifts or adversarial attacks and typically require model fine-tuning. To address this, we propose a plug-and-play safety patching framework that operates without modifying the original model’s weights. Our method introduces learnable, data-driven safety-aware conditioning signals into intermediate layers of the frozen diffusion model during the denoising process. It supports multi-strategy fusion and cross-model transferability, significantly enhancing resilience against both distribution shifts and adversarial prompts. Evaluated on six state-of-the-art T2I models, our approach reduces unsafe image generation to 7%, outperforming seven existing SOTA safety methods, while preserving high-fidelity image quality and strong text–image alignment.

Existing LLM safety evaluations predominantly rely on binary (safe/unsafe) classification, failing to capture the nuanced risk gradient of extremist content. Method: We introduce XGUARD, a benchmark built upon 3,840 real-world, extremist-related red-teaming prompts, featuring a fine-grained, five-level danger scale (0–4). We propose a novel tiered safety evaluation framework and the Attack Severity Curve (ASC)—an interpretable metric enabling risk modeling and cross-intensity comparison of defense strategies. Our methodology integrates social-media- and news-driven prompt engineering, multi-level human annotation, visualized evaluation, and lightweight defense validation. Results: Experiments across six mainstream LLMs and two defense approaches reveal systemic ideological safety gaps; moreover, model robustness exhibits a significant trade-off with expressive capability.

Existing LLM safety guardrails suffer from limited real-time performance, inadequate multimodal support, and poor interpretability. To address these gaps, this paper proposes a native multimodal safety guardian system designed for enterprise deployment. The system unifies processing of text, image, and audio inputs, employing category-specific LoRA adapters and a teacher-assisted reasoning-chain annotation pipeline to establish a four-dimensional safety framework: toxicity, gender bias, data privacy, and prompt injection. Leveraging efficient LoRA fine-tuning and a curated multimodal safety dataset, the system delivers real-time, auditable, and interpretable compliance enforcement. Extensive evaluation demonstrates significant improvements over WildGuard, LlamaGuard-4, and GPT-4.1 across multiple safety benchmarks, achieving state-of-the-art performance among both open-source and proprietary models—making it suitable for highly regulated production environments.

This work addresses the limitations of existing abuse detection methods, which rely on static models and manual annotations and struggle to handle dynamic, context-sensitive online abusive behaviors. It proposes the first large language model (LLM)-integrated framework spanning the entire lifecycle of abuse detection, systematically encompassing label and feature generation, detection, appeal review, and audit governance. Bridging academic research and industrial practice, the study explores LLMs’ capabilities in contextual reasoning, policy interpretation, and cross-modal understanding. The authors delineate key architectural considerations for each phase, evaluate LLMs’ potential and limitations regarding interpretability, policy alignment, and multimodal fusion, and identify critical challenges—including latency, cost, determinism, adversarial robustness, and fairness—thereby offering a principled direction toward building reliable, accountable, large-scale abuse governance systems.

While text-driven 3D editing can generate high-fidelity, multi-view consistent scenes, it is susceptible to propagating coherent NSFW content throughout the 3D representation when prompted with unsafe inputs. To address this issue, this work proposes 3DEditSafe, the first framework to systematically mitigate such risks by directly imposing safety constraints during the optimization of 3D Gaussian splatting. The approach integrates multiple mechanisms—including safety-guided generation, safety-aware regularization on rendered views, semantic safety projection, residual suppression, and mask-aware preservation—into the 3D representation refinement process. Experiments demonstrate that 3DEditSafe significantly reduces unsafe semantic alignment and view-level attack success rates on EditSplat, while also revealing an inherent trade-off between safety enforcement and output fidelity.

The proliferation of toxic text generated by large language models (LLMs) undermines the robustness of toxicity classifiers and increases their susceptibility to adversarial attacks. Method: This paper proposes a mechanistic interpretability–driven active defense framework: it introduces attention-head-level circuit analysis—the first such application—to diagnose classifier vulnerabilities; integrates fine-grained attribution with adversarial attack localization to identify critical, attack-prone components; and enhances robustness via targeted circuit suppression. Contribution/Results: Evaluated on BERT and RoBERTa architectures across diverse demographic datasets, the method significantly improves classification accuracy under adversarial perturbations. It further uncovers systematic differences in model vulnerability across demographic groups, revealing fairness-related failure modes. By unifying interpretability, robustness, and fairness, this work establishes a novel paradigm for building trustworthy, auditable, and attack-resilient content moderation systems.

This study investigates whether existing large-model-based commercial image moderation systems are robust against evasion attacks employing simple image transformations. It presents the first systematic evaluation of three leading APIs under seven black-box image manipulations—such as color inversion and grayscale conversion—that require no gradients, surrogate models, or internal system knowledge. The findings reveal that even fixed transformations easily interpretable by humans can significantly bypass these moderation systems, with particularly pronounced vulnerabilities in multimodal content and self-harm categories. These results challenge the feasibility of relying solely on large-model APIs as standalone security boundaries and underscore their insufficiency for constructing dependable content safety mechanisms.

This study investigates the impact of unsafe image proportions in training data on the safety of text-to-image generative models. By constructing controlled datasets and training multiple model variants with contamination rates ranging from 0% to 9.6% while holding other factors constant, the authors evaluate output safety using four independent safety classifiers, ablation studies of text encoders (including SafeCLIP), and quality metrics such as FID, CLIPScore, and ImageReward. They reveal, for the first time, a monotonic dose–response relationship between training contamination level and output unsafety. Notably, even with zero contamination, a baseline risk of 16.6% persists, which SafeCLIP reduces to 9.6% without compromising generation quality. These findings demonstrate that the text encoder itself constitutes an inherent safety risk, offering a new perspective for safer model design.

This work addresses critical limitations in existing static benchmarks for harmful content detection—namely, their constrained scalability, limited diversity, and susceptibility to contamination from pretraining corpora. To overcome these issues, the authors propose a dynamic synthesis framework grounded in role-based simulation. By constructing two-dimensional user profiles that integrate demographic attributes with topical interests, the framework guides large language models to generate contextualized, highly harmful, and diverse content. The approach leverages role-guided agents, human-in-the-loop evaluation, and multidimensional analysis to produce data that significantly surpasses current benchmarks in terms of harmfulness, challenge level, and diversity. The resulting dataset exerts stronger stress on mainstream detection systems and achieves linguistic and thematic diversity comparable to manually curated datasets.