Adaptive Tabu Dropout for Regularization of Deep Neural Networks

2024-12-31

International Conference on Neural Information Processing

AI Summary

To address overfitting and insufficient training diversity caused by fixed dropout rates in deep neural networks, this paper proposes Adaptive Tabu Dropout (ATD). ATD innovatively introduces tabu search principles into regularization: it dynamically constructs a tabu set of weights to suppress updates of highly important parameters; adaptively adjusts the dropout rate based on gradient sensitivity; and estimates parameter importance via Hessian approximation. This enables joint optimization of dropout rate and parameter importance. Experiments on CIFAR-10, CIFAR-100, and an ImageNet subset demonstrate that ATD consistently improves generalization accuracy by 1.2–2.7% over baseline methods—including standard dropout and DropBlock—while enhancing training stability. The key contribution lies in the first integration of tabu search heuristics into stochastic regularization, enabling context-aware, importance-driven neuron deactivation during training.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Problem

• Deep Neural Networks
• Improved Dropout Method
• Overfitting

Innovation

• Tabu Dropout
• Variable Tabu Period
• Self-adjusting Algorithm