Thoughts on Objectives of Sparse and Hierarchical Masked Image Model

2025-05-12

AI Summary

This work investigates how masking patterns affect the self-supervised pretraining performance of SparK, revealing that conventional random masking struggles to jointly model local details and global hierarchical structures. To address this, we propose Structured Mesh Masking: an image is partitioned into multi-scale grids, and tokens are masked in a hierarchical, coarse-to-fine manner—enabling joint optimization of sparsity and hierarchy. We integrate Mesh Masking into the SparK framework, jointly optimizing hierarchical feature reconstruction and contrastive representation learning. On ImageNet-1K linear evaluation, our method achieves a +1.8% top-1 accuracy gain over the baseline. To our knowledge, this is the first work to incorporate explicit grid-based geometric structure into masking design, demonstrating that mask geometry serves as a critical inductive bias for visual representation quality. Our approach establishes a new paradigm for sparse, hierarchical self-supervised learning.

Technology Category

Computer Vision: Representation Learning for VisionMachine Learning: Structured LearningSearch and Optimization: Learning to Search

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphsWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

Masked image modeling is one of the most poplular objectives of training. Recently, the SparK model has been proposed with superior performance among self-supervised learning models. This paper proposes a new mask pattern for this SparK model, proposing it as the Mesh Mask-ed SparK model. We report the effect of the mask pattern used for image masking in pre-training on performance.

Problem

• Improving masked image modeling objectives
• Enhancing SparK model performance
• Evaluating mask pattern impact

Innovation

• New mask pattern for SparK model
• Mesh Mask-ed SparK model introduced
• Studies mask pattern impact on performance