AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility

2026-10-07

AI Summary

This study addresses the prohibitive retraining costs incurred when scaling instruction-tuned models to new domains within dynamic, heterogeneous environments. To this end, we propose a taxonomy-free modular framework that automatically discovers latent domains and trains independent LoRA adapters in parallel. By integrating a parameter-independent routing mechanism, our approach constructively eliminates inter-domain interference at the architectural level, thereby enabling cost-effective modular domain expansion. Experimental results demonstrate that the proposed framework achieves performance comparable to full fine-tuning across 14 benchmarks while facilitating the efficient integration of new domains without retraining. Ultimately, this work provides a scalable and lightweight paradigm for the continual adaptation of large language models.

Abstract

Instruction-tuned models are deployed into environments where domains are heterogeneous and evolve, yet adding new domains or data typically requires costly retraining. We present AutoAdapt, a modular framework that incorporates new domains and data via targeted single-adapter training without modifying other adapters. The framework automatically discovers latent domains, uses them to train per-domain Low-Rank Adaptation (LoRA) adapters independently in parallel and performs parameter-free routing. Across 14 domain-specific benchmarks and GPT-4o pairwise judgements, AutoAdapt achieves parity with a LoRA adapter trained on all domains without requiring full-model retraining. We also find evidence of specialisation effect convergence across independent discovery methods. Overall, training each adapter on its own domain prevents domain interference by construction, thus enabling modular, taxonomy-free domain specialisation without aggregate performance loss or full model retraining.

Problem

• instruction-tuned models
• domain adaptation
• low-rank adaptation
• modular extensibility
• domain interference

Innovation

• AutoAdapt
• Low-Rank Adaptation (LoRA)
• Automatic Domain Discovery
• Parameter-free Routing
• Modular Framework