Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models

2026-09-30

AI Summary

This study addresses the routing collapse issue in Mixture-of-Experts (MoE) architectures for time-series foundation models, caused by statistical information stripping during instance normalization. To overcome this, we propose RR-MoA, a causal intervention mechanism grounded in mutual information decomposition that reveals the underlying causes of normalization-induced degradation. Specifically, this work pioneers leveraging raw inputs for routing decisions at the pre-normalization stage, combined with a frozen-backbone adapter mixture technique to enable efficient adaptation across heterogeneous data. Our approach transcends conventional MoE optimization bottlenecks, empirically validates the "freezing paradox," and demonstrates strong cross-backbone generalizability. Extensive evaluation across 54 comparative experiments shows that RR-MoA consistently outperforms both LoRA and full-parameter fine-tuning, achieving state-of-the-art performance without exception.

Abstract

Time series foundation models (TSFMs) commonly adapt to new data by attaching a single trainable head to a frozen backbone, a one-size-fits-all setup that underfits heterogeneous regimes. Replacing the head with a mixture of experts is the standard upgrade, but on instance-normalized backbones (the dominant TSFM design class) it fails: routing entropy collapses to zero and one expert absorbs every input, a failure we call normalization-induced routing collapse. Standard MoE rescue mechanisms do not repair it, because the cause is in the router's input, not its optimization. Pre-encoder normalization strips the statistics a router would need to tell regimes apart. A mutual-information decomposition makes this precise and yields a signal-ratio that, computed before training, predicts dataset vulnerability (Spearman $ρ= -0.88$). Eight causal controls, including a vision-modality replication, isolate instance normalization as the cause. The prescription is a minimal causal intervention: Raw-Routed Mixture of Adapters (RR-MoA), which routes on the raw, pre-normalization input. Under a strictly frozen backbone, RR-MoA wins 54/54 comparisons against the strongest fixed adapter and significantly outperforms LoRA, TRACE, AdaMix, and full fine-tuning. The effect generalizes across six backbones and an imputation task. Frozen RR-MoA also beats full fine-tuning by 12-79% (the Frozen Paradox); two architecturally distinct variants confirm the principle generalizes beyond this specific router.

Problem

• Time Series Foundation Models
• Mixture of Experts
• Routing Collapse
• Instance Normalization
• Adapter

Innovation

• Routing Collapse
• Causal Intervention
• Mixture of Adapters
• Time Series Foundation Models
• Instance Normalization