MPO: Multilingual Safety Alignment via Reward Gap Optimization

2025-05-22

AI Summary

Existing safety alignment methods—such as RLHF and DPO—are predominantly monolingual and suffer significant performance degradation on noisy multilingual data, hindering the global safe deployment of large language models (LLMs). To address this, we propose a novel cross-lingual reward gap optimization paradigm anchored on English, introducing the first preference-learning-based reward modeling framework coupled with a cross-lingual reward discrepancy minimization mechanism. This enables lossless transfer of safety capabilities from English to multiple languages. Our approach integrates multilingual safety instruction tuning with a unified evaluation framework. Empirical results on LLaMA-3.1, Gemma-2, and Qwen2.5 demonstrate substantial improvements in safety alignment across eight languages, while preserving original language understanding and generation capabilities—without compromising multilingual utility.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Large language models (LLMs) have become increasingly central to AI applications worldwide, necessitating robust multilingual safety alignment to ensure secure deployment across diverse linguistic contexts. Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data. To address these limitations, we introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages. MPO directly minimizes the reward gap difference between the dominant language and target languages, effectively transferring safety capabilities while preserving the original strengths of the dominant language. Extensive experiments on three LLMs, LLaMA-3.1, Gemma-2 and Qwen2.5, validate MPO's efficacy in multilingual safety alignment without degrading general multilingual utility.

Problem

• Ensures robust multilingual safety alignment for LLMs
• Addresses noisy multilingual data in preference learning methods
• Transfers English safety capabilities to other languages effectively

Innovation

• Optimizes reward gap for multilingual safety
• Transfers English safety to other languages
• Preserves original language strengths effectively