🤖 AI Summary
This study investigates the marginal benefits of a three-stage strategy—self-supervised pretraining, intermediate fine-tuning, and downstream task adaptation—for small-scale Vision Transformers (~5M parameters). Motivated by the observation that intermediate fine-tuning may degrade downstream performance due to task misalignment, we propose a systematic ablation framework to assess the impact of varying dataset and objective combinations across stages. Experiments reveal that targeted pretraining substantially improves small-model performance, whereas introducing semantically distant intermediate tasks yields no gain—and often harms performance while wasting compute. The core contribution is the empirical demonstration that, for small ViTs, **the quality of data selection is far more critical than the number of stacked tasks**, challenging conventional assumptions about multi-stage transfer. This finding provides key empirical evidence and methodological guidance for designing efficient, lightweight self-supervised learning paradigms.
📝 Abstract
While transformer-based architectures have taken computer vision and NLP by storm, they often require a vast amount of parameters and training data to attain strong performance. In this work, we experiment with three distinct pre-training, intermediate fine-tuning, and downstream datasets and training objectives to explore their marginal benefits on a small 5M-parameter vision transformer. We find that while pre-training and fine-tuning always help our model but have diminishing returns, intermediate fine-tuning can actually show harmful impact on downstream performance, potentially due to dissimilarity in task mechanics. Taken together, our results suggest that small-scale ViTs benefit most from targeted pre-training and careful data selection, while indiscriminate stacking of intermediate tasks can waste compute and even degrade performance.