🤖 AI Summary
To address overfitting in over-parameterized neural networks, this paper proposes an adaptive regularization method grounded in feature consistency: it extracts features from multiple random subsets of the training set and explicitly enforces representational consistency across independent and identically distributed (i.i.d.) subsets by minimizing pairwise L2 or cosine distances among them. This is the first approach to model overfitting as *inconsistent representations on i.i.d. data subsets*, requiring no architectural or task-specific assumptions—making it plug-and-play. The method introduces zero gradient modifications and adds no extra learnable parameters. Experiments demonstrate that it significantly reduces the train–validation loss gap and improves generalization accuracy across diverse network architectures and tasks. Moreover, it exhibits strong hyperparameter robustness and negligible computational overhead.
📝 Abstract
Over-parameterized neural network models often lead to significant performance discrepancies between training and test sets, a phenomenon known as overfitting. To address this, researchers have proposed numerous regularization techniques tailored to various tasks and model architectures. In this paper, we introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets. Based on this viewpoint, we propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set. Due to minimal prior assumptions, this approach is applicable to almost any architecture and task. Our experiments show that it effectively reduces overfitting, with low sensitivity to hyperparameters and minimal computational cost. It demonstrates particularly strong memory suppression and promotes normal convergence, even when the model has already started to overfit. Even in the absence of significant overfitting, our method consistently improves accuracy and reduces validation loss.