SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization

2026-08-04

AI Summary

This work investigates whether the performance bottleneck of Mixture-of-Experts (MoE) networks under fixed total parameter budgets stems from the routing algorithm itself or from the alignment between training signals and the granularity of category partitioning. To this end, the authors propose SpecDrop, a parameter-free, fixed routing mechanism that assigns dominant weights to a primary expert based on input class labels while incorporating minor leakage weights, followed by fixed normalized output aggregation (requiring class labels at inference). Experiments demonstrate that routing efficacy hinges critically on the alignment between module structure and category granularity—not routing complexity. SpecDrop achieves 79.23% and 79.89% top-1 accuracy on CIFAR-100 and ImageNet-1K, respectively, substantially outperforming non-routing baselines, yet degrades to baseline performance under ambiguous category partitions, confirming the pivotal role of granularity alignment.

Abstract

Mixture-of-experts (MoE) networks pursue specialization through learned routers, gates, and load-balancing losses, yet at matched total-parameter budgets learned routers can underperform equal-weight No-Routing baselines. Is the bottleneck the routing algorithm, or the alignment between training-signal granularity and the target categories? We probe the question with SpecDrop, a fixed parameter-free routing scheme: each of $K$ branches receives weight $p_a$ for its assigned category and a small leakage $p_i > 0$ otherwise, merged through a category-independent fixed denominator, with no learned routing parameters and no auxiliary losses; the category label is required at inference. On vision tasks where each image has one superclass label (CIFAR-100 on ResNet-110; ImageNet-1K on ViT-S/16), SpecDrop reaches 79.23% on CIFAR-100 and 79.89% on ImageNet-1K, exceeding parameter-matched baselines that do not use the label (+4.75 over dense on CIFAR-100; +6.53 over the No-Routing+SE control on ImageNet-1K). These gains quantify what category supervision buys when deployed through routing -- not an advantage over label-aware deployments of the baselines: given the same label, masking a dense model's outputs is stronger for accuracy alone (85.2 / 83.7). SpecDrop's contribution is converting the label into trained-in modular structure: 58%/100% branch-category alignment, and masking gains of 0.00 (CIFAR) / +1.06 (ImageNet) -- the output-space restriction is largely internalized during training. On fuzzy partitions, where training units span multiple categories (SlimPajama-6B language modeling with a 30M Transformer; SuperNI instruction tuning over Llama-3.2-1B with LoRA), the routing mechanism reduces to the matched No-Routing controls within seed noise, the null our thesis predicts. Granularity alignment, not algorithm choice, localizes when routing helps. Code: https://github.com/Beryex/SpecDrop

Problem

• Mixture-of-Experts
• routing
• category alignment
• modular specialization
• parameter budget

Innovation

• Mixture-of-Experts
• parameter-free routing
• category-conditioned specialization
• modular architecture
• granularity alignment