MMAligner: Safeguarding Multimodal Large Language Models through Representation Calibration

2026-08-06

AI Summary

This work addresses a critical vulnerability in multimodal large language models (MLLMs), which often fail to reject semantically equivalent unsafe multimodal inputs due to safety representation shift. The study reveals, for the first time, that this degradation stems from inaccurate internal representations rather than an inherent lack of safety capabilities. To mitigate this issue, the authors propose a lightweight, data-efficient representation calibration method grounded in geometric analysis of the safety subspace. By integrating hard lower-bound and soft upper-bound constraints with fidelity-preserving optimization on benign inputs, the method effectively maps unsafe inputs back into the model’s existing rejection region. Extensive experiments demonstrate that the approach achieves a 99% rejection rate on unsafe inputs across multiple open-source MLLMs while incurring less than 2% utility loss, significantly outperforming current baselines.

Abstract

Multimodal large language models (MLLMs) often refuse unsafe text prompts yet generate harmful responses to semantically equivalent multimodal inputs. Existing defenses either rely on external guardrails, which add inference overhead without repairing intrinsic flaws, or safety fine-tuning, which treats alignment as black-box optimization and may sacrifice utility or require large multimodal datasets. To identify the cause of this safety disparity, we analyze MLLM representations geometrically. We find that safety mechanisms learned from text persist across modalities: a shared safety subspace and refusal boundary remain effective, and representations inside this boundary consistently trigger refusals. However, unsafe multimodal inputs undergo a representation shift that places most of them outside the boundary, allowing them to bypass the model's intrinsic safety mechanism. This indicates that multimodal safety degradation stems from representation misalignment rather than the absence of safety capability. Based on this finding, we propose MMAligner, a safeguarding method that calibrates unsafe multimodal representations into the pre-existing refusal region. MMAligner applies a hard lower bound to ensure refusal, a soft upper bound to avoid excessive modification, and a preservation objective for benign inputs. Experiments across multiple open-source MLLMs show that MMAligner raises the average refusal rate on unsafe multimodal inputs to 99% with less than 2% utility degradation and minimal training data, substantially improving the safety-utility trade-off over existing baselines. (*Due to the notification from arXiv, "The Abstract field cannot be longer than 1,920 characters", the Abstract that appeared is shortened.)

Problem

• Multimodal Large Language Models
• Safety Alignment
• Representation Shift
• Refusal Mechanism
• Multimodal Safety

Innovation

• representation calibration
• multimodal safety
• refusal boundary
• safety alignment
• geometric analysis