🤖 AI Summary
To address overfitting in Transformer-based text classification models, this paper proposes a lightweight stochastic token masking regularization technique: during training, each input token is replaced with [MASK] with probability $p = 0.1$, introducing controllable input perturbation. Theoretically, we first model this method as implicit gradient averaging and implicit ensemble, revealing its dual mechanism—simultaneously mitigating overfitting and strengthening inter-token dependencies. Crucially, the approach requires no architectural modifications or additional parameters. Extensive cross-model validation on mBERT, Qwen2.5-0.5B, and TinyLlama-1.1B demonstrates consistent superiority over standard regularizers (e.g., Dropout, Label Smoothing) across language identification and sentiment analysis tasks, yielding average accuracy gains of 1.2–2.8 percentage points. The method exhibits strong generalizability and robustness across diverse model scales and linguistic tasks.
📝 Abstract
While transformer-based models achieve strong performance on text classification, we explore whether masking input tokens can further enhance their effectiveness. We propose token masking regularization, a simple yet theoretically motivated method that randomly replaces input tokens with a special [MASK] token at probability p. This introduces stochastic perturbations during training, leading to implicit gradient averaging that encourages the model to capture deeper inter-token dependencies. Experiments on language identification and sentiment analysis -- across diverse models (mBERT, Qwen2.5-0.5B, TinyLlama-1.1B) -- show consistent improvements over standard regularization techniques. We identify task-specific optimal masking rates, with p = 0.1 as a strong general default. We attribute the gains to two key effects: (1) input perturbation reduces overfitting, and (2) gradient-level smoothing acts as implicit ensembling.