Do ResNets Route? Sparse Interaction Experts in Residual Networks

2026-10-02

AI Summary

This study investigates how Residual Networks (ResNets) dynamically allocate functional contributions across modules for different inputs and whether implicit routing mechanisms exist internally. By modeling module compositions as set functions, this work leverages Möbius inversion theory to precisely decompose ResNet outputs, quantifying higher-order interaction terms among modules. Through extensive analysis of ImageNet-pretrained models, the research examines the sparsity and input-dependence of these interactions. The findings reveal that higher-order interaction magnitudes concentrate at specific orders, with dominant patterns shifting dynamically according to the input, thereby uncovering an implicit soft routing mechanism within dense ResNets. This mechanism enables interaction-expert-based functional allocation without requiring explicit routers, offering novel insights into conditional computation in deep networks.

Abstract

Residual networks execute every block for every input, yet their functional contributions need not be input independent. We formulate a trained ResNet as a set function over binary residual-branch masks and apply Möbius inversion to decompose its output exactly into individual residual corrections and higher-order interactions. For smooth residual stacks, we show that each fixed $k$-way interaction scales as $\mathcal{O}(λ^k)$ under residual scaling. Exhaustive analysis of ImageNet-pretrained ResNet-18 and ResNet-34 reveals that interaction mass peaks at orders five and ten, respectively, rather than at low orders. Reducing the residual scale shifts both spectra toward lower orders, but also changes model predictions. The interaction coefficients are concentrated in magnitude but not hard sparse, and prediction-preserving sparsity weakens with depth. Crucially, the dominant interactions vary across inputs and predicted classes around a shared global core, while their overall complexity changes little with sample difficulty. These results show that dense ResNets implement an implicit form of soft routing: every block is executed, but different inputs rely on different residual interaction experts. Routing can therefore emerge at the level of functional contribution without an explicit router or sparse execution.

Problem

• Residual Networks
• Soft Routing
• Interaction Experts
• Möbius Inversion
• Set Function

Innovation

• Residual Networks
• Möbius Inversion
• Soft Routing
• Sparse Interaction Experts
• Set Function