receptive field control

Existing parameter-efficient fine-tuning (PEFT) methods struggle to match the performance of full fine-tuning in dense prediction tasks due to input-agnostic modeling and redundant cross-layer representations. This work proposes AdaRoute, a dynamic parameter routing mechanism based on a shared expert pool. During forward propagation, AdaRoute dynamically generates input-adaptive low-rank weights for each module, enabling customized feature representation. By sharing experts across layers, it facilitates implicit feature interaction and enhances representational diversity. Integrating concepts from mixture-of-experts (MoE), dynamic routing, and low-rank adaptation, AdaRoute significantly outperforms existing PEFT approaches across a range of dense prediction tasks, including semantic segmentation, object detection, instance segmentation, and panoptic segmentation.

This work addresses the limitation of existing Mixture-of-Experts (MoE)-based parameter-efficient fine-tuning (PEFT) methods, which struggle to simultaneously capture high-level semantics and fine-grained syntactic requirements due to their neglect of the hierarchical complexity inherent in tasks. To overcome this, we propose Expert Pyramid Tuning (EPT), a novel architecture that, for the first time, integrates a multi-scale feature pyramid mechanism into the PEFT framework. EPT generates multi-scale features through a shared meta-knowledge subspace and pyramid projections, dynamically composing them via a task-aware router. By synergistically combining LoRA, MoE, and learnable up-projection operators, EPT establishes a two-stage tuning pipeline and supports post-training reparameterization for parameter compression. Extensive experiments demonstrate that EPT significantly outperforms current MoE-LoRA approaches across multiple multitask benchmarks while reducing the number of trainable parameters.

This work systematically investigates the effectiveness of sparse Top-k routing in image classification, addressing limitations of Sparse Mixture-of-Experts (MoE) models such as expert collapse and marginal end-to-end efficiency gains. The study reveals that performance improvements critically depend on the routing computation ratio (ρ) and the activation of multiple experts (k ≥ 2), while also highlighting the computational leverage provided by the backbone network. To mitigate batch scheduling inefficiencies, the authors propose a per-sample variant of Soft MoE and validate their approach through hard capacity-constrained routing, hidden dimension scaling, and ablation studies on ImageNet-1K. The method consistently outperforms baselines across CIFAR-10/100, Tiny-ImageNet, and ImageNet-1K, with the per-sample Soft MoE achieving superior accuracy over dense counterparts on CIFAR-100.

To address excessive computational overhead when deploying vision models on resource-constrained devices, this paper proposes an efficient Vision Transformer (ViT) architecture design framework. Methodologically: (1) it optimizes the input-output data pathway to enhance representational capacity of lightweight models; (2) it restructures the context window of computationally constrained attention mechanisms to improve local-global modeling efficiency; and (3) it leverages the invertibility and explicit probabilistic modeling properties of normalizing flows to enable high-fidelity, low-overhead knowledge distillation. Experiments demonstrate that the proposed approach achieves comparable or superior accuracy on benchmarks such as ImageNet, while requiring significantly fewer parameters and FLOPs. It also substantially reduces inference latency and memory footprint. The framework establishes a scalable new paradigm for efficient visual understanding at the edge.

This work addresses the limitations of existing parameter-efficient fine-tuning methods, such as LoRA, which rely on fixed architectures and struggle in dynamic scenarios involving task conflicts or sensor failures. The authors propose a novel low-rank mixture-of-experts architecture that integrates local spatial partitioning with context-aware dynamic routing, available in both block-level and unit-level variants. A key innovation is a decentralized unit-level gating mechanism that achieves performance approaching ideal global routing without requiring centralized coordination. By combining block-wise low-rank decomposition, dynamically sparse routing, and a matrix-grid coordinate field model, the method establishes gradient firewalls to suppress error propagation. Experiments on high-dimensional matrix simulation, tabular data transformation, and vision tasks under sensor degradation demonstrate substantial improvements over static baselines, achieving robust and efficient dynamic adaptation.

Existing research on vision-based Mixture-of-Experts (MoE) models predominantly relies on category-level routing statistics, which obscures the actual representational content encoded by individual experts. This work trains sparsely gated convolutional MoE models and advances expert analysis from categorical labels to continuous visual and semantic feature dimensions for the first time. By integrating contrastive learning, neuroscience-inspired tuning analyses, and representational similarity analysis (RSA)—augmented with human semantic judgments from the THINGS dataset to define semantic axes—we demonstrate that experts consistently differentiate along continuous semantic dimensions such as “animate–inanimate.” Despite sparse routing, experts collectively span a broad semantic space. While experts exhibit comparable category discriminability, their feature tuning profiles differ markedly, underscoring the necessity and efficacy of expert-level representational analysis.

This work addresses the challenge of efficiently integrating the complementary capabilities of multiple vision foundation models (VFMs) while mitigating negative transfer and feature interference. The authors propose PRISM, a dual-stream mixture-of-experts (MoE) framework that employs a two-stage paradigm: first, experts self-organize and specialize within decoupled representation subspaces; subsequently, they are dynamically recombined to construct sparse, task-specific computational pathways for downstream tasks. PRISM introduces, for the first time, a self-organized expert specialization mechanism that operates without manual intervention, synergistically combining teacher-conditioned routing with dynamic routing strategies to substantially alleviate interference in multi-model fusion. Experiments demonstrate that PRISM achieves new state-of-the-art performance on PASCAL-Context and NYUD-v2, underscoring the scalability and effectiveness of sparse, specialized pathways in harnessing diverse visual knowledge.

Training high-capacity vision models is computationally expensive, and existing model-growing approaches rely on pre-trained narrow models, failing to accurately reflect the total training cost. This work proposes a Recursive Block-Diagonal Coupling (RBDC) training protocol that recursively integrates independently trained narrow models into a high-performance wide model via a parameter-free block-diagonal structure. The method incurs no additional parameters and substantially reduces training FLOPs—achieving a 30% reduction in computational cost on ImageNet compared to standard from-scratch training while maintaining comparable accuracy. Under identical computational budgets, RBDC outperforms current model-growing strategies and serves as a superior backbone, enhancing performance in downstream tasks such as object detection and instance segmentation.

This work addresses the routing collapse and expert deadlocks that commonly afflict Token-Choice sparse Mixture-of-Experts (MoE) architectures in video diffusion Transformers, which severely limit expert diversity utilization. Starting from a 5-billion-parameter dense model, the authors formulate three principles for converting dense networks to MoE. Through temporal routing analysis of 65 million tokens, they reveal that deadlocked layers follow a U-shaped distribution across the network depth and propose a “functional redundancy” hypothesis to explain this phenomenon. Building on these insights, they integrate expert cloning, zero-initialized gating, auxiliary losses, and enhanced router designs—including linear, MLP, and cross-attention variants—to effectively mitigate bfloat16 precision pitfalls. Their approach alleviates single-expert deadlocks in approximately two-thirds of network layers, endows the model with partial self-recovery capability, delineates the capacity limits of the Token-Choice paradigm, and outlines a three-stage roadmap toward unified vision models and ultimately world models.

This work addresses the fundamental trade-off in sparse Mixture-of-Experts (MoE) models between load balancing and expert specialization, which often leads to routing collapse or diminished expert diversity. The authors propose Hi-MoE, a novel framework that decomposes routing into two coupled hierarchical levels: inter-group routing ensures balanced token distribution across expert groups, while intra-group routing fosters complementary expert specialization and prevents collapse. This principled redesign of router behavior consistently outperforms existing sparse routing and grouped MoE approaches across both NLP and vision benchmarks. In a 58B-token pretraining setting, Hi-MoE-7B achieves a 5.6% lower perplexity and 40% improved expert balance compared to OLMoE-7B.