🤖 AI Summary
This work addresses cross-domain interference in multi-domain adapter composition for large language models by proposing the DoRA-RBAC hierarchical composition framework and systematically comparing Euclidean averaging with Riemannian geometry–based directional normalization fusion strategies. Through experiments on multiple question-answering benchmarks, the study provides the first empirical evidence that orthogonality and angular alignment of parameter updates are not reliable predictors of adapter composition performance, thereby challenging the prevailing assumption that the geometric structure of parameter space primarily governs interference. The results demonstrate that, although single-domain performance matches that of LoRA, geometry-aware fusion does not significantly outperform standard averaging in multi-domain settings, suggesting that interference likely arises from interactions within shared nonlinear representations rather than parameter-space geometry.
📝 Abstract
Access control in large language models (LLMs) requires modular mechanisms to enable domain-specific behavior without retraining or cross-domain interference. A common hypothesis is that interference during adapter composition arises from overlap in linear parameter updates, suggesting that enforcing orthogonality or directional independence should improve multi-domain performance. We test this hypothesis using DoRA-RBAC, a hierarchical adapter composition framework based on weight-decomposed low-rank adaptation. We compare conventional Euclidean merging with a geometry-aware Riemannian-inspired merging strategy that approximates the Frechet mean via normalized directional averaging across multiple QA benchmarks (GPQA, PubMedQA, SimpleQA, WMDP) on LLaMA-3.1-8B and Mistral-7B. Our results show that while single-domain performance matches LoRA, geometry-aware merging provides no consistent advantage over standard averaging in multi-domain settings.Diagnostic analysis further reveals that angular alignment and orthogonality of adapter updates are weak predictors of composition performance. These findings suggest that adapter interference is not governed primarily by parameter-space geometry, but is instead consistent with interactions in shared nonlinear representations.