Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing

2026-05-30

AI Summary

This work addresses the over-conservatism of conventional large language model alignment methods, which often discard unsafe data and consequently produce uninformative responses to sensitive yet harmless queries. The authors propose a dialectical alignment paradigm that deliberately retains and controllably integrates domain knowledge from unsafe corpora instead of simply filtering it out. They introduce the first alignment framework that trains domain-specific experts on unsafe data while employing safe samples to guide dynamic routing via a Mixture-of-Experts architecture. This approach combines LoRA-based expert modules with a lightweight safety gating network to enable context-aware, secure response generation. Experiments demonstrate that the model achieves a relative improvement of over 20% (absolute gain exceeding 15%) in safe response rate on stringent safety benchmarks, substantially enhances answer informativeness, and exhibits strong zero-shot generalization across domains.

Abstract

The prevailing paradigm in large language model (LLM) alignment operates via erasure, filtering unsafe data or training models to strictly refuse harmful prompts. While effective at reducing immediate toxicity, this approach fundamentally constricts the model's epistemological scope, resulting in over-cautious systems that output uninformative blanket refusals to sensitive yet benign queries. In this work, we challenge the orthodoxy that unsafe data must be discarded. We propose a dialectical approach to alignment, positing that unsafe data encodes rich, domain specific knowledge critical for nuanced, safe, and informative generation. To operationalize this, we introduce SafeMoE, a Mixture-of-Experts (MoE) framework that isolates unsafe knowledge into domain-specific Low-Rank Adapters (LoRA experts) trained exclusively on harmful corpora. To synthesize safety from these unsafe primitives, we train a lightweight gating network using a minimal, highly curated set of safe-informative responses. During inference, this router dynamically orchestrates the unsafe experts, effectively steering the generation trajectory to harness their deep domain knowledge while strictly enforcing safety constraints. Extensive empirical evaluations across stringent safety benchmarks demonstrate that SafeMoE is not only safer, achieving over a 20% relative improvement in safe response rate (more than a 15% absolute gain), but also produces more informative responses when safety and harmfulness are of paramount concern. Furthermore, the routing mechanism exhibits strong zero-shot generalization to unseen domains and broader safety tasks without domain-specific supervision. Our findings suggest a paradigm shift in alignment: true safety requires not the masking of unsafe knowledge, but its controlled integration.

Problem

• alignment
• unsafe knowledge
• large language models
• safety
• informative responses

Innovation

• SafeMoE
• Mixture-of-Experts
• LoRA
• dynamic safety routing
• alignment