Problem
Research questions and friction points this paper is trying to address.
self-supervised learning
masked autoencoders
representation learning
transfer learning
fine-grained tasks
Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning
🤖 AI Summary
This study addresses the limited feature transferability and generalization of Masked Autoencoders (MAE) in self-supervised learning. To overcome this, we propose an improved architecture based on dual-view augmentation and independent encoding. The core contribution is a novel “Swingers” strategy that exchanges global representations (CLS tokens) across views prior to decoding, compelling the model to learn view-invariant and robust semantic features. This approach effectively integrates masked modeling with cross-view representation interaction. Experimental results demonstrate that the proposed method improves ImageNet kNN classification accuracy by 3–5%, significantly enhances performance on fine-grained tasks, and reduces state probing error by 64%.
📝 Abstract
Self-supervised learning (SSL) removes the need for annotations and makes models that are capable across more domains than supervised learning. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches. In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image to facilitate efficient transfer. We perform extensive experiments, and find Masked Swingers outperforms MAE by +3-5% on ImageNet-1K kNN and provides large gains on fine-grained tasks, e.g., relative gains of +45% on instance retrieval, +22% on animal re-ID, and +76% on Omniglot character recognition. To boot, Swingers reduces error -64% relative to MAE on three new state-probing datasets, opening the door to world modeling. Welcome to our Swingers party.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Masked Autoencoders
Self-Supervised Learning
Data Augmentation
Token Swapping
View-Agnostic Representation
🔎 Similar Papers
No similar papers found.
