🤖 AI Summary
Existing methods struggle to provide clear and generalizable explanations of self-supervised vision models, particularly in distinguishing behavioral differences between contrastive learning and masked image modeling. This work proposes a visualization protocol based on unsupervised semantic segmentation, introduced for the first time as an interpretability tool, to intuitively reveal positional bias, locality bias, and scaling properties across different layers and representations of such models. The approach effectively disentangles positional effects from local structural patterns and uncovers novel phenomena—such as boundary artifacts in DINOv3-Large—offering a unified, intuitive, and reproducible visual framework for understanding diverse self-supervised pretraining paradigms.
📝 Abstract
Self-supervised learning (SSL) has produced a diverse landscape of vision transformers (ViTs) whose pretrained representations support a wide range of downstream tasks. Towards a better understanding of these models, a body of work has assessed the mechanics of their self-attention as well as the types of information captured across their representations, revealing, for example, stark differences between models trained with contrastive learning (CL) and masked image modeling (MIM). However, these advances in model understanding have not yet fully permeated the broader community, where insights specific to CL models are sometimes generalized to MIM models. To make model understanding straightforward and intuitive for a broad audience, we propose a simple and easily interpretable visualization protocol.
Our protocol is based on visualizing unsupervised semantic segmentation results, yet our goal is not to maximize segmentation performance. Instead, it allows us to convey model behaviors that consistently emerge across images. Benchmarking a diverse set of SSL models across layers and representations, we obtain novel insights into distinct positional biases and scaling behaviors, including strong boundary artifacts in DINOv3-Large model tokens. These insights complement and help communicate a range of previous findings.
Our protocol further enables a clear visual distinction between positional effects and the closely related but distinct locality bias, which has been studied much more extensively in the literature. The protocol is publicly available on GitHub and we believe it will catalyze further model understanding for a broad community.