Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting

2026-07-28

AI Summary

This work addresses the trade-off between intervention placement and response strategies in content moderation systems, noting that conventional component-level evaluations often fail to capture real user experience. The authors propose evaluating moderation configurations through end-to-end user-centric metrics—namely usefulness and harmful exposure—and systematically compare input filtering, response filtering, and combined hard-blocking strategies. They further introduce large language model–based rewriting and probe routing mechanisms. Experimental results demonstrate that response filtering substantially improves usefulness, while joint filtering effectively reduces harmful exposure. Rewriting recovers blocked but benign traffic without increasing harmful content, and probe routing significantly lowers conditional generation latency. The study underscores the necessity of end-to-end evaluation and offers practical guidance for deploying moderation strategies in real-world scenarios.

Abstract

Content-moderation classifiers are usually evaluated in isolation, but deployment requires choosing where to intervene and what follows a flag. We evaluate these choices using two end-to-end customer-outcome metrics rather than component accuracy: Usefulness, the fraction of turns with a shown, non-harmful, relevant response, and Harmful Exposure, the fraction with a shown harmful response. Latency and error rates are diagnostics. We compare Input only, Response only, and Input + response hard blocking on a human-labelled product benchmark and public ToxicChat evaluation. At the evaluated operating points, Response only achieves the highest filter-only Usefulness in both settings, while Input + response achieves lower Harmful Exposure. Replacing Response only blocking with Response + rewrite recovers most blocked traffic and yields the same observed Harmful Exposure count as Response only blocking for the selected configuration; this equality is not an equivalence result. Probe routing substantially reduces conditional route-and-generation time relative to LLM routing at comparable measured outcomes. A focused output review shows how rewrites balance filter passage with usefulness by generalizing triggering language while retaining benign intent and safe redirection; some sensitive-domain outputs nevertheless omit potentially safety-relevant support information. These results support comparing moderation configurations under deployment-specific safety and latency constraints rather than applying a universal placement rule. Code and public artifacts are available at https://github.com/microsoft/mod-frontier

Problem

• content moderation
• filter placement
• response rewriting
• harmful exposure
• usefulness

Innovation

• content moderation
• end-to-end evaluation
• response rewriting
• filter placement
• probe routing