Problem
Research questions and friction points this paper is trying to address.
text-to-image generation
safety control
foundation models
modular steering
energy-based sampling
Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models
🤖 AI Summary
This work addresses the critical challenge of achieving safe and controllable text-to-image generation without compromising output quality. The authors propose a training-free, inference-stage energy-guidance framework that leverages a frozen vision-language foundation model as a plug-and-play semantic energy estimator. By utilizing gradient feedback from this estimator, the method enables multi-objective safety control. Compatible with both diffusion and flow-matching models, the approach integrates latent-space clean estimation with energy-based sampling. It demonstrates state-of-the-art robustness against NSFW red-teaming attacks while preserving high-fidelity image generation for benign prompts.
📝 Abstract
Controlling the behavior of text-to-image generative models is critical for safe and practical deployment. Existing safety approaches typically rely on model fine-tuning or curated datasets, which can degrade generation quality or limit scalability. We propose an inference-time steering framework that leverages gradient feedback from frozen pretrained foundation models to guide the generation process without modifying the underlying generator. Our key observation is that vision-language foundation models encode rich semantic representations that can be repurposed as off-the-shelf supervisory signals during generation. By injecting such feedback through clean latent estimates at each sampling step, our method formulates safety steering as an energy-based sampling problem. This design enables modular, training-free safety control that is compatible with both diffusion and flow-matching models and can generalize across diverse visual concepts. Experiments demonstrate state-of-the-art robustness against NSFW red-teaming benchmarks and effective multi-target steering, while preserving high generation quality on benign non-targeted prompts. Our framework provides a principled approach for utilizing foundation models as semantic energy estimators, enabling reliable and scalable safety control for text-to-image generation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
energy-based sampling
inference-time steering
foundation models
training-free safety
modular control
🔎 Similar Papers
No similar papers found.
