SEQR: Secure and Efficient QR-based LoRA Routing

2025-09-22

AI Summary

Unsupervised, secure, and efficient routing of LoRA adapters in large language models remains an open challenge. Method: This paper formally defines the unsupervised LoRA routing objective and proposes SEQR—a novel algorithm that dynamically selects optimal adapters based solely on input activation norm maximization, requiring no supervision, fine-tuning, or additional training. SEQR integrates QR decomposition with low-rank activation analysis to rigorously guarantee routing correctness and enable plug-and-play dynamic adapter composition. Contribution/Results: Extensive experiments demonstrate that SEQR significantly outperforms existing routing methods across multi-task benchmarks. It maintains or improves downstream task performance while reducing inference overhead by 30–50%. The approach achieves high accuracy, strong scalability, and deployment security—enabling efficient, trustworthy adapter orchestration without compromising model integrity or requiring retraining.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Low-Rank Adaptation (LoRA) has become a standard technique for parameter-efficient fine-tuning of large language models, enabling large libraries of LoRAs, each for a specific task or domain. Efficiently selecting the correct LoRA adapter for a given input remains a challenge, particularly in secure environments where supervised training of routers may raise privacy concerns. Motivated by previous approaches, we formalize the goal of unsupervised LoRA routing in terms of activation norm maximization, providing a theoretical framework for analysis. We demonstrate the discriminative power of activation norms and introduce SEQR, an unsupervised LoRA routing algorithm designed to maximize efficiency while providing strict routing guarantees. SEQR provably identifies the norm-maximizing adapter with significantly greater efficiency, making it a highly scalable and effective solution for dynamic LoRA composition. We validate our results through experiments that demonstrate improved multi-task performance and efficiency.

Problem

• Unsupervised routing of LoRA adapters without supervised training
• Secure LoRA selection in privacy-sensitive environments lacking labels
• Efficient identification of optimal adapter using activation norm maximization

Innovation

• Unsupervised LoRA routing via activation norm maximization
• SEQR algorithm provides strict routing guarantees
• Efficiently identifies norm-maximizing adapter for scalability