Ray-Tracing for Conditionally Activated Neural Networks

2025-02-20

AI Summary

This work addresses parameter redundancy in neural network inference by proposing a condition-driven dynamic sparsity architecture. Methodologically, it introduces a hierarchical Mixture of Experts (MoE) framework integrated with a ray-tracing-inspired progressive sampling mechanism, enabling input-complexity-aware expert path activation and adaptive network expansion. Crucially, it is the first to embed convergence-aware sampling into MoE training, achieving complexity-adaptive sparsification without explicit regularization. The contributions are threefold: (1) a novel sparse paradigm that jointly orchestrates conditional activation and dynamic structural expansion; (2) path-specific backpropagation to preserve gradient efficacy across activated subnetworks; and (3) empirical results demonstrating competitive accuracy with dense models on image classification tasks, while substantially reducing inference parameter count—where compression ratio naturally increases with input complexity.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

In this paper, we introduce a novel architecture for conditionally activated neural networks combining a hierarchical construction of multiple Mixture of Experts (MoEs) layers with a sampling mechanism that progressively converges to an optimized configuration of expert activation. This methodology enables the dynamic unfolding of the network's architecture, facilitating efficient path-specific training. Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines while significantly reducing the parameter count required for inference. Notably, this parameter reduction correlates with the complexity of the input patterns, a property naturally emerging from the network's operational dynamics without necessitating explicit auxiliary penalty functions.

Problem

• Optimizes expert activation in neural networks
• Reduces parameter count for inference
• Enables efficient path-specific training

Innovation

• Hierarchical Mixture of Experts
• Dynamic network architecture unfolding
• Reduced inference parameters efficiently