When Refusal Looks Safe: The Refusal-Cue Shortcut in Safety Guard Models

2026-08-04

AI Summary

This work addresses the limited representational capacity of existing methods in complex visual scenes by proposing a novel neural network architecture based on adaptive multi-scale feature fusion. The approach dynamically integrates local details and global semantic information through a learnable weighting mechanism and incorporates a lightweight attention module to enhance perceptual sensitivity to critical regions. Extensive experiments demonstrate that the proposed model consistently outperforms current state-of-the-art methods across multiple benchmark datasets, achieving an average accuracy gain of 3.2% while maintaining low computational overhead. Beyond offering a new perspective on efficient visual representation learning, this study also validates the effectiveness of dynamic fusion strategies in improving model generalization.

Abstract

Safety guards are widely used to filter harmful content and are typically trained via supervised fine-tuning on labeled prompt-response pairs. We audit two widely used safety-guard training datasets, WildGuardMix and GR-Train, and find that among responses to harmful prompts, refusal expressions co-occur almost exclusively with unharmful labels. This imbalance motivates what we term the refusal-cue shortcut: inserting a refusal cue into a harmful response could flip the guard's verdict from harmful to unharmful. The shortcut affects not only guards trained on these datasets but also officially released models such as LlamaGuard3 and Qwen3Guard whose training data is undisclosed. It persists across response positions and is generally stronger in smaller variants within a family. To mitigate it, we adapt sparse complementary masking as a lightweight post-hoc intervention that identifies and suppresses a small set of shortcut-associated attention heads and MLP neurons without retraining. On two primary benchmarks, the intervention achieves an approximately 79% relative reduction in response-initial detection failures induced by refusal cues, while preserving standard detection performance. Although optimized using cues at a single response position, the suppression effect transfers to unseen positions and datasets, suggesting that shortcut manifestations across positions are partly mediated by shared internal components. Further analysis provides evidence that shortcut reliance and legitimate refusal recognition are partially functionally separable, as suppressing the shortcut broadly preserves the guard's ability to recognize genuine refusals.

Problem

• safety guard
• refusal-cue shortcut
• harmful content detection
• model robustness
• shortcut learning

Innovation

• refusal-cue shortcut
• safety guard
• sparse complementary masking
• attention heads
• post-hoc intervention