lora diffusion adaptation

Facing the challenge of inefficient navigation and filtering among over 100,000 LoRA adapters hosted on large-scale platforms, this paper proposes a submodular optimization-based adapter selection framework. It formulates adapter retrieval as a combinatorial optimization problem balancing relevance and diversity. Methodologically, we design a differentiable submodular objective function that jointly incorporates low-rank decomposition features from attention layers and semantic similarity metrics, enabling end-to-end optimization. We evaluate the approach quantitatively (retrieval accuracy, diversity score) and qualitatively (generation quality, stylistic coverage) on text-to-image generation tasks. Experiments demonstrate significant improvements across multiple domains: +12.3% Recall@10 in adapter retrieval and +28.6% LPIPS diversity gain, while preserving generation fidelity. This work establishes a novel paradigm for efficient utilization of large-scale lightweight adapter repositories.

Existing LoRA adapters require retraining when the base model changes and rely on original or high-quality synthetic data, severely limiting cross-model generalization. This paper proposes the first training-free, data-free cross-model LoRA transfer method: by enforcing subspace alignment constraints and layer-wise weight similarity filtering, it enables zero-shot mapping and reuse of LoRA parameters across distinct base models (e.g., Stable Diffusion v1.5 and SDXL). The approach comprises three core components: low-rank parameter projection, inter-layer similarity measurement, and architecture-aware alignment. Experimental results demonstrate that, without access to any original or synthetic data, the transferred adapters retain over 98% of the original LoRA’s generation performance. This significantly enhances the portability and practical utility of LoRA adapters across diverse diffusion architectures.

This work addresses a central challenge in parameter-efficient fine-tuning: identifying the optimal placement of adapters to achieve peak performance with minimal parameters. The authors propose PAGE, a metric based on initial gradient energy analysis, which reveals that adaptation effects are highly concentrated in the down-projection modules of shallow feed-forward networks. Leveraging this insight, they introduce DomLoRA—a method that deploys a single LoRA adapter exclusively in this dominant module. This study is the first to demonstrate the existence, architectural dependency, and task stability of such a dominant adaptation module, establishing a new paradigm for efficient fine-tuning. Experiments show that DomLoRA, using only ~0.7% of the parameters of standard LoRA, consistently outperforms it across diverse tasks—including instruction following, mathematical reasoning, code generation, and multi-turn dialogue—and further enhances the effectiveness of other LoRA variants.

To address the high power consumption, low efficiency, and memory bottlenecks in mobile LoRA fine-tuning of large diffusion models, this work proposes the first full-precision quantized training acceleration architecture specifically designed for LoRA fine-tuning. The architecture introduces a high-utilization dataflow supporting irregular tensor shapes, integrating full-precision quantized training, a customized LoRA-specific dataflow, and hardware-software co-optimization for low-rank adaptation. Experiments demonstrate 1.81× training speedup and 5.50× energy efficiency improvement over baseline implementations, with negligible degradation in image generation quality (ΔFID < 0.3). The core contribution lies in the first end-to-end integration of full-precision quantized training and LoRA-aware hardware acceleration, establishing a scalable paradigm for efficient diffusion model fine-tuning on edge devices.

To address overfitting, poor generalization, and limited generation diversity in single-image fine-tuning of diffusion models, this paper proposes a timestep-dependent Low-Rank Adaptation (LoRA) framework. Our method dynamically modulates the rank constraint across timesteps and incorporates orthogonal initialization to ensure parameter independence of the adapters, effectively mitigating overfitting—particularly at high timesteps. Compared to standard LoRA and other personalization approaches, our method achieves a superior trade-off between concept fidelity and text–image alignment. Empirically, it maintains strong generalization and high-fidelity generation even under extreme data scarcity (e.g., only one input image), while remaining computationally efficient. The framework is thus well-suited for real-world deployment where both training data and computational resources are severely constrained.

本文提出LoRango方法，通过Signature和Payload适配器解决多LoRA组合时的安全隐患，实现对文本到图像扩散模型的配对条件攻击。

This work addresses the risk of training data memorization in diffusion models fine-tuned with Low-Rank Adaptation (LoRA), which poses significant copyright and privacy concerns—particularly when only LoRA weights are shared, rendering existing mitigation strategies ineffective. To tackle this challenge, the authors propose Base-Anchored Filtering (BAF), a post-processing framework that operates without access to original training data or additional retraining. BAF leverages spectral decomposition to map LoRA updates into channel space and selectively preserves generalizable features while suppressing memorized components based on their alignment with the principal subspace of the pre-trained backbone model. Notably, BAF achieves memory reduction using only the LoRA weights themselves. Extensive experiments across multiple datasets and diffusion architectures demonstrate that BAF substantially mitigates memorization risks while maintaining or even enhancing generation quality.

研究通过控制实验分析了不同LoRA秩在扩散模型微调中的性能与成本平衡，结果表明中等秩（如4或8）在保持高质量的同时具有最高的效率。

This work addresses the limitation of fixed-rank constraints in parameter-efficient fine-tuning, which fail to accommodate the heterogeneous rank requirements across different layers of neural networks. The authors propose LR-LoRA, a novel approach that introduces a learnable rank mechanism within the LoRA framework, enabling differentiable and dynamic optimization of the rank for each adapter layer. This method reveals a systematic disparity in rank demands between attention and MLP layers in Transformers, thereby providing a more flexible and effective inductive bias. Experimental results demonstrate that LR-LoRA significantly outperforms existing parameter-efficient fine-tuning methods across multiple benchmarks for language understanding and commonsense reasoning, achieving state-of-the-art performance.

This study addresses the significant performance gap between Low-Rank Adaptation (LoRA) and full fine-tuning. To bridge this disparity, we propose GDLoRA, which introduces a novel orthogonal decomposition mechanism based on the reachable gradient space of LoRA. Specifically, our method extracts the orthogonal component of the gradient to directly update the base model weights. By integrating the AdamW optimizer with forward and backward signal reconstruction techniques, GDLoRA achieves effective optimization of base parameters without incurring additional memory overhead. Extensive experiments demonstrate that GDLoRA significantly outperforms standard LoRA across natural language understanding and mathematical reasoning tasks, substantially narrowing the performance gap with full fine-tuning. This work establishes a new paradigm for parameter-efficient fine-tuning.