Technology Category
Computer Vision: Adversarial Attacks & RobustnessMachine Learning: Adversarial Learning & RobustnessNatural Language Processing: Safety and Robustness
Robust Representation Learning in Masked Autoencoders
🤖 AI Summary
This study investigates the intrinsic mechanisms underlying the robustness of Masked Autoencoders (MAE) to image degradations such as blur and occlusion in classification tasks. Through layer-wise analysis of token embeddings, combined with subspace separability assessment and global attention visualization, the authors find that MAE establishes persistent global attention early in the encoder and progressively enhances class separability with depth. To quantify this robustness, they introduce two novel metrics: directional alignment between clean and perturbed embeddings, and head-level retention rate of active features under degradation. The results demonstrate that MAE-learned latent representations maintain high classification performance despite image corruption, offering both theoretical insight and quantitative tools for understanding its robustness.
Technology Category
Application Category
📝 Abstract
Masked Autoencoders (MAEs) achieve impressive performance in image classification tasks, yet the internal representations they learn remain less understood. This work started as an attempt to understand the strong downstream classification performance of MAE. In this process we discover that representations learned with the pretraining and fine-tuning, are quite robust - demonstrating a good classification performance in the presence of degradations, such as blur and occlusions. Through layer-wise analysis of token embeddings, we show that pretrained MAE progressively constructs its latent space in a class-aware manner across network depth: embeddings from different classes lie in subspaces that become increasingly separable. We further observe that MAE exhibits early and persistent global attention across encoder layers, in contrast to standard Vision Transformers (ViTs). To quantify feature robustness, we introduce two sensitivity indicators: directional alignment between clean and perturbed embeddings, and head-wise retention of active features under degradations. These studies help establish the robust classification performance of MAEs.
Problem
Research questions and friction points this paper is trying to address.
Masked Autoencoders
robust representation
image classification
feature robustness
representation learning
Innovation
Methods, ideas, or system contributions that make the work stand out.
Masked Autoencoders
Robust Representation Learning
Class-aware Latent Space
Global Attention
Feature Sensitivity Metrics
🔎 Similar Papers
No similar papers found.
