Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed

2026-07-27

AI Summary

This study addresses the limitations of general-purpose text-to-image toxicity detectors, which—due to their reliance on broad safety standards—fail to identify harmful content targeting marginalized groups such as people with disabilities, misclassifying approximately 35% of such instances as safe. To remedy this, the authors propose a Community-specific Toxicity Detection (CTD) framework that collaborates with disability community experts to develop tailored safety guidelines. Evaluated on a dataset of 2,400 annotated images depicting individuals with dwarfism and visual impairments, the work provides the first systematic demonstration of the necessity and feasibility of community-adapted toxicity detection. Leveraging large vision-language models with in-context learning, VQA-style prompting, and parameter-efficient fine-tuning, the approach adapts to community-specific norms using fewer than 100 examples. Experiments show that off-the-shelf models achieve zero-shot F1 scores of only 0.32–0.37; optimized GPT-4o reaches 0.50–0.78, and fine-tuned smaller models attain up to 0.59—still substantially below performance on generic tasks, highlighting the challenge of domain adaptation.

Abstract

State-of-the-art toxicity detectors for text-to-image generation adopt a one-size-fits-all approach: a single universal model applying fixed safety guidelines to all users. Our empirical evidence shows that these detectors fail to shield marginalized communities: approximately 35% of generated images labeled safe are considered harmful by disability communities. In this position paper, we argue for community-specific toxicity detection (CTD). To demonstrate its feasibility, we collaborate with disability experts to develop safety guidelines for two communities: dwarfism and blind/low vision. Using a dataset of 2,400 annotated T2I-generated images we demonstrate that both large vision-language models and existing general-purpose toxicity detectors catastrophically fail to recognize harmful content under these guidelines in zero-shot settings with F1 score lower than random guessing (F1 0.32 and 0.37). Promisingly, prompt-based adaptation methods (ICL, VQA) substantially improve harm detection performance (GPT-4o: F1 0.50 and 0.78), while parameter-efficient fine-tuning improves smaller models (0.5b-7b with best F1 0.48 and 0.59) with less than 100 demonstrations, but remains sensitive to evolving guidelines. Despite these gains, CTD performance remains far below F1 $\approx 0.9$ achieved for general-purpose toxicity detection, highlighting the challenge and the need for sustained research effort.

Problem

• toxicity detection
• text-to-image generation
• community-specific harm
• marginalized communities
• safety guidelines

Innovation

• community-specific toxicity detection
• text-to-image generation
• safety guidelines
• prompt-based adaptation
• parameter-efficient fine-tuning