MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation

2026-08-04

AI Summary

This work addresses the high storage overhead and limited instance-level adaptability of existing Mixture-of-Experts (MoE)-based parameter-efficient fine-tuning (PEFT) methods, which typically require storing multiple full LoRA experts. To overcome these limitations, the authors propose MoEGen, a novel framework that reinterprets the MoE mechanism by shifting from discrete expert selection to continuous expert conditioning. Specifically, MoEGen employs compact learnable expert codes whose weighted combinations drive a lightweight hypernetwork to dynamically generate input-specific low-rank adaptation parameters. This design effectively decouples expert capacity from storage cost, enabling efficient instance-level adaptation while maintaining model compactness. Extensive experiments demonstrate that MoEGen significantly outperforms both static and MoE-based PEFT baselines across eight commonsense reasoning benchmarks and joint tasks in specialized domains such as medicine and law.

Abstract

Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of large language models, but existing MoE-based PEFT methods typically improve capacity by storing multiple full LoRA experts, causing adapter storage to grow linearly with the number of experts and restricting adaptation to a fixed expert pool. We ask whether MoE-based PEFT can produce instance-specific adaptations without explicitly storing a separate LoRA module for each expert. To address this gap, we propose MoEGen, an adaptation framework that shifts MoE-based PEFT from expert selection to expert-conditioned parameter generation. Instead of storing each expert as a full LoRA adapter, MoEGen represents each expert as a small learnable vector, termed an expert code. It routes each input over these vectors and uses their weighted combination to condition a lightweight hypernetwork that generates input-specific low-rank updates. This design decouples expert capacity from adapter storage while enabling instance-conditioned adaptation. Experiments on eight commonsense reasoning benchmarks show consistent improvements over strong static and MoE-based PEFT baselines across three backbones. MoEGen also performs strongly in joint medical and legal-domain adaptation.

Problem

• Mixture-of-Experts
• Parameter-efficient fine-tuning
• LoRA
• instance-adaptive adaptation
• adapter storage

Innovation

• Mixture-of-Experts
• Parameter-Efficient Fine-Tuning
• LoRA
• Hypernetwork
• Instance-Adaptive Adaptation