Configurable Reward Model for Balanced Safety Alignment

2026-05-28

AI Summary

Large language models struggle to generalize across heterogeneous and dynamically evolving safety alignment requirements. To address this challenge, this work proposes the Configurable Safety Reward Model (CSRM), which, for the first time, enables fine-grained safety configuration and context-sensitive reward modeling. CSRM jointly optimizes safety compliance calibration with reward learning and introduces a configuration-guided data augmentation strategy that preserves the relative severity structure while enhancing instruction-following capabilities. Without requiring additional human annotations, CSRM achieves state-of-the-art performance on CoSApien (94.6% F1) and DynaBench (75.8% F1), significantly improving generalization to unseen safety configurations and enabling downstream alignment models to achieve a better trade-off between helpfulness and safety.

Abstract

Aligning large language models (LLMs) to heterogeneous and rapidly evolving safety requirements remains a critical challenge. Existing instruction-tuned LLMs and standalone safety classifiers often fail to generalize to new safety configurations, motivating the need for Reward Models (RMs) that are explicitly configurable to changing specifications. We introduce the Configurable Safety Reward Model (CSRM), which is jointly optimized for calibrated safety compliance and reward modeling. Our approach is supported by configuration-targeted data augmentation that enforces instruction adherence while preserving relative severity structure. The resulting RM is sensitive to fine-grained safety configurations and conversational nuances, substantially improving generalization to previously unseen safety configurations. CSRM achieves state-of-the-art performance on recent configurable safety benchmarks, including CoSApien (94.6% F1) and DynaBench (75.8% F1), without requiring additional human annotation. When used for downstream safety alignment, CSRM yields LLMs with a significantly improved helpfulness-safety tradeoff compared to existing baselines.

Problem

• safety alignment
• configurable reward model
• large language models
• safety requirements
• generalization

Innovation

• Configurable Reward Model
• Safety Alignment
• Data Augmentation
• Reward Modeling
• Generalization