LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models

2026-09-22

AI Summary

本文提出LoRango方法，通过Signature和Payload适配器解决多LoRA组合时的安全隐患，实现对文本到图像扩散模型的配对条件攻击。

Abstract

Users commonly combine multiple Low-Rank Adaptation (LoRA) adapters to personalize images with different subjects, styles, and visual attributes. Yet inspecting adapters individually does not establish the safety of their composition. We identify and characterize a pair-conditioned attack in text-to-image diffusion: individually useful and benign-appearing adapters redirect image generation when co-loaded with a specifically matched partner, whose identity serves as the trigger. We introduce LoRango to realize this attack through complementary Signature and Payload adapters. The Signature writes a pair-specific code into intermediate carrier representations, while the Payload uses code-selective responses and opposing signal/reference branches. These branches approximately cancel for standalone adapters and mismatched pairs; matched code-reader alignment breaks cancellation within native GEGLU blocks and releases the programmed action. Both adapters are exported as ordinary static LoRA files compatible with standard loaders, requiring no prompt trigger or base-pipeline modification. LoRango achieves matched-pair attack success rates of 97.9\% on SD v1.5 and 98.7\% on SDXL, compared with 2.8--4.6\% when implanted adapters are loaded individually. Further experiments evaluate pair selectivity, standalone fidelity, robustness to deployment variations, and applicability across denoiser architectures. These findings show that individual-adapter inspection is insufficient to assess the security of multi-LoRA personalization and motivate auditing adapter compositions.

Problem

• Low-Rank Adaptation
• Diffusion Models
• Security
• Adapter Composition
• Text-to-Image

Innovation

• Low-Rank Adaptation
• pair-conditioned attack
• Signature and Payload adapters
• code-selective responses
• matched code-reader alignment