Do Models Share Safety Representations? Cross-Model Steering for Safe Visual Generation

2026-06-03

AI Summary

This work proposes the first cross-model safety steering framework for generative models, addressing the limitation of existing safety control methods that rely on unsafe data from the target model and thus lack transferability. By learning safety directions from paired safe/unsafe prompts in a source model and aligning them to the target model using only benign data, the approach enables inference-time safety control without requiring unsafe examples from the target. The method introduces a multi-vector mechanism for category-level fine-grained protection and leverages both CLIP semantic space and generative latent space to facilitate direction transfer. Experiments on text-to-image and text-to-video generation demonstrate that the transferred safety directions effectively reduce attack success rates (ASR) while preserving generation quality (measured by CLIP-Score and FID), achieving performance comparable to locally trained safety methods.

Abstract

Recent progress in generative modeling has made safety control a central challenge, yet existing approaches remain largely model-specific, requiring retraining or tailored interventions for each new architecture. In this work, we ask whether safety can be represented as a portable latent direction, learned once and reused across heterogeneous generators. We introduce the first framework for cross-model safety steering, in which a safety direction is estimated in a source LLM from paired safe-unsafe prompts, transported to a target generator through a lightweight alignment fitted on benign data alone, and applied at inference time. Crucially, our pipeline never accesses unsafe data on the target side, isolating whether safety can be transferred through shared representation geometry. Beyond a single global direction, we also identify a multi-vector extension that captures category-specific safety behaviors, enabling more selective control. We evaluate our approach in text-to-image and text-to-video generation across diverse source-target model pairs. Across models, transferred safety directions achieve ASR reduction and CLIP-Score/FID trade-offs comparable to directions learned natively on the target model using unsafe data, while requiring no target-side unsafe data. This indicates that safety improvements do not come at the expense of generation quality. Our results point to a modular view of safety: safety-relevant behavior is not purely model-local, but can be controlled through latent directions that persist across models. This suggests a new path toward lightweight, reusable safety mechanisms that do not require target-side unsafe data.

Problem

• safety representation
• cross-model transfer
• generative modeling
• latent direction
• safe visual generation

Innovation

• cross-model safety steering
• safety representation
• latent direction
• generative modeling
• transferable safety