Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models

2026-03-07

AI Summary

Current safety alignment of large language models relies on costly, static human annotations and red-teaming evaluations, which struggle to adapt to model evolution and often compromise utility. This work proposes Self-MOA, a framework that establishes the first fully automated safety alignment loop: it employs weak supervision to automatically evaluate the model, dynamically generates targeted red-team prompts, self-constructs preference data, and applies multi-objective preference optimization to simultaneously enhance both safety and usefulness. Evaluated across multiple small language models and safety benchmarks, Self-MOA achieves a 12.41% improvement in safety while preserving model utility, using only approximately one-eleventh of the human supervision data required by conventional approaches.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Safety alignment is critical for deploying large language models (LLMs) in real-world applications, yet most existing approaches rely on large human-annotated datasets and static red-teaming benchmarks that are costly, difficult to scale, and slow to adapt to evolving model behaviors. Moreover, overly conservative safety mechanisms can reduce model usefulness by rejecting sensitive but legitimate queries. We introduce Self-MOA (Self Multi-Objective Alignment), a fully automated framework for aligning small language models using weak supervision from automated evaluator models. Self-MOA operates as a closed loop that dynamically generates model-specific red team prompts, constructs preference data from model-generated responses, and aligns models via multi-objective preference optimization to jointly optimize for safety and helpfulness. Across multiple small language models and safety benchmarks, Self-MOA achieves a 12.41\% improvement in safety while preserving helpfulness, using as little as 11 times less training data than human-supervised alignment baselines. These results demonstrate that adaptive, automated alignment can reduce the dependence on static, human-curated safety pipelines in resource-constrained settings.

Problem

• safety alignment
• weak supervision
• red-teaming
• helpfulness
• language models

Innovation

• weak supervision
• automated alignment
• multi-objective optimization
• red teaming
• small language models