Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation

2026-03-12

AI Summary

This work addresses the limitation of existing Mixture-of-Experts (MoE)-based parameter-efficient fine-tuning (PEFT) methods, which struggle to simultaneously capture high-level semantics and fine-grained syntactic requirements due to their neglect of the hierarchical complexity inherent in tasks. To overcome this, we propose Expert Pyramid Tuning (EPT), a novel architecture that, for the first time, integrates a multi-scale feature pyramid mechanism into the PEFT framework. EPT generates multi-scale features through a shared meta-knowledge subspace and pyramid projections, dynamically composing them via a task-aware router. By synergistically combining LoRA, MoE, and learnable up-projection operators, EPT establishes a two-stage tuning pipeline and supports post-training reparameterization for parameter compression. Extensive experiments demonstrate that EPT significantly outperforms current MoE-LoRA approaches across multiple multitask benchmarks while reducing the number of trainable parameters.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has become a dominant paradigm for deploying LLMs in multi-task scenarios due to its extreme parameter efficiency. While Mixture-of-Experts (MoE) based LoRA variants have achieved promising results by dynamically routing tokens to different low-rank experts, they largely overlook the hierarchical nature of task complexity. Existing methods typically employ experts with uniform architectures, limiting their ability to capture diverse feature granularities required by distinct tasks--where some tasks demand high-level semantic abstraction while others require fine-grained syntactic manipulation. To bridge this gap, we propose Expert Pyramid Tuning (EPT), a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT. Unlike standard LoRA, EPT decomposes task adaptation into two stages: (1) A shared meta-knowledge Subspace that encodes universal linguistic patterns in low dimensions; (2) A Pyramid Projection Mechanism that utilizes learnable up-projection operators to reconstruct high-dimensional features at varying scales. A task-aware router then dynamically selects the optimal combination of these multi-scale features. Extensive experiments across multiple multi-task benchmarks demonstrate that EPT significantly outperforms SOTA MoE-LoRA variants. Crucially, thanks to the re-parameterization capability of our design, EPT achieves this performance improvement while simultaneously reducing the number of training parameters.

Problem

• Parameter-Efficient Fine-Tuning
• Mixture-of-Experts
• task complexity hierarchy
• feature granularity
• multi-task learning

Innovation

• Expert Pyramid Tuning
• Parameter-Efficient Fine-Tuning
• Mixture-of-Experts
• Multi-scale Feature Pyramid
• Task-aware Routing