Norm or Direction? Decoding Vision Mambas for High-Resolution Vision

2026-07-20

AI Summary

This study investigates the root cause of performance disparities between Vision Mamba and MambaOut in high-resolution vision tasks. Through cross-model CKA analysis, Grad-CAM visualizations, and a novel token norm–direction disentanglement method, the work reveals for the first time that Vision Mamba primarily encodes class-discriminative information in token directions rather than norms, whereas MambaOut relies predominantly on token norms. This direction-dominated mechanism substantially enhances stability and performance in dense prediction tasks such as semantic segmentation. Full fine-tuning experiments demonstrate that Vision Mamba significantly outperforms MambaOut in semantic segmentation, underscoring the critical role of token directional structure in high-resolution visual representation learning.

Abstract

Vision Mamba models replace quadratic self-attention with linear complexity selective state space models (SSMs), emerging as efficient visual backbones. However, MambaOut demonstrates that a Gated CNN block can match or exceed VMamba on image classification, questioning the necessity of SSMs for vision. This raises a fundamental question: do VMamba and MambaOut encode visual information differently at the representation level? To investigate, we apply cross model centered kernel alignment (CKA) analysis and find that VMamba's final stage blocks form representations distinctly different from both MambaOut and its own preceding blocks. We therefore focus on the final block features, decomposing each spatial token into magnitude and direction. MambaOut concentrates class-discriminative information in high-norm foreground tokens that align with Grad-CAM attribution. VMamba, by contrast, produces high-norm tokens predominantly in background regions, misaligned with Grad-CAM, yet preserves discriminative signals primarily in token directions. These observations reveal that the two models rely on different encoding strategies. We connect this difference to high-resolution classification and semantic segmentation. VMamba distributes logit support broadly across object regions, whereas MambaOut relies on sparse dominant tokens, a strategy that becomes less stable as token counts grow. Under full fine-tuning for segmentation, VMamba consistently outperforms MambaOut. These results suggest that VMamba's advantage in dense prediction stems not merely from the SSM mechanism or sequence length, but from how semantic evidence is organized across token magnitude, direction. Ultimately, we conclude that token magnitude and directional structure serve as critical axes for improving visual backbones, particularly under dense supervision.

Problem

• Vision Mamba
• representation analysis
• token norm
• token direction
• dense prediction

Innovation

• Vision Mamba
• token direction
• magnitude-direction decomposition
• dense prediction
• representation analysis