QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation

2025-07-06

AI Summary

To address content-style feature entanglement arising from multi-LoRA fusion in text-to-image models, this paper proposes QR-LoRA—a parameter-efficient fine-tuning framework based on QR decomposition. It decomposes weight updates into an orthogonal matrix **Q**, which suppresses cross-attribute interference, and an upper-triangular matrix **R**, which encodes attribute-specific transformations; a learnable increment ΔR enables clean, contamination-free merging of multiple tasks. Compared to standard LoRA, QR-LoRA reduces parameter count by 50% while inherently satisfying orthogonality constraints. Experiments demonstrate substantial improvements in both content-style disentangled generation and secure multi-model fusion—achieving higher generation quality and disentanglement accuracy—without compromising efficiency or controllability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Existing text-to-image models often rely on parameter fine-tuning techniques such as Low-Rank Adaptation (LoRA) to customize visual attributes. However, when combining multiple LoRA models for content-style fusion tasks, unstructured modifications of weight matrices often lead to undesired feature entanglement between content and style attributes. We propose QR-LoRA, a novel fine-tuning framework leveraging QR decomposition for structured parameter updates that effectively separate visual attributes. Our key insight is that the orthogonal Q matrix naturally minimizes interference between different visual features, while the upper triangular R matrix efficiently encodes attribute-specific transformations. Our approach fixes both Q and R matrices while only training an additional task-specific $ΔR$ matrix. This structured design reduces trainable parameters to half of conventional LoRA methods and supports effective merging of multiple adaptations without cross-contamination due to the strong disentanglement properties between $ΔR$ matrices. Experiments demonstrate that QR-LoRA achieves superior disentanglement in content-style fusion tasks, establishing a new paradigm for parameter-efficient, disentangled fine-tuning in generative models.

Problem

• Disentangle content-style attributes in LoRA fine-tuning
• Reduce parameter count in generative model adaptation
• Prevent cross-contamination when merging multiple LoRA models

Innovation

• QR decomposition for structured parameter updates
• Orthogonal Q matrix minimizes feature interference
• Trainable ΔR matrix reduces parameters by half