Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models

2025-07-21

AI Summary

This work addresses a performance bottleneck in low-parameter (<1.5M) deep vision models caused by feature map interference—specifically, the aberrant encoding of multiple semantic concepts by individual neurons. We identify a previously unrecognized interference mechanism: excessive neuronal responses induced by the synergistic effect of bottleneck architectures and superlinear activation functions. To mitigate this, we propose the NoDepth Bottleneck architecture, which decouples channel compression from nonlinear transformation and incorporates a gradient-aware interference suppression module to explicitly constrain feature superposition. Systematic evaluation on ImageNet demonstrates that our approach significantly improves scaling efficiency and accuracy for compact models—for instance, boosting Top-1 accuracy by 2.3% over a ResNet-18 variant. Notably, this is the first method to enable controllable modeling of interference under superlinear activations, establishing a novel design paradigm for ultra-lightweight vision models.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The paper investigates the performance of state-of-the-art low-parameter deep neural networks for computer vision, focusing on bottleneck architectures and their behavior using superlinear activation functions. We address interference in feature maps, a phenomenon associated with superposition, where neurons simultaneously encode multiple characteristics. Our research suggests that limiting interference can enhance scaling and accuracy in very low-scaled networks (under 1.5M parameters). We identify key design elements that reduce interference by examining various bottleneck architectures, leading to a more efficient neural network. Consequently, we propose a proof-of-concept architecture named NoDepth Bottleneck built on mechanistic insights from our experiments, demonstrating robust scaling accuracy on the ImageNet dataset. These findings contribute to more efficient and scalable neural networks for the low-parameter range and advance the understanding of bottlenecks in computer vision. https://caiac.pubpub.org/pub/3dh6rsel

Problem

• Investigates performance of low-parameter vision models
• Addresses interference in feature maps for better accuracy
• Proposes NoDepth Bottleneck to enhance scaling efficiency

Innovation

• Uses bottleneck architectures with superlinear activations
• Reduces interference in low-parameter feature maps
• Proposes NoDepth Bottleneck for scalable accuracy