mask-based representation learning

This work investigates how masking patterns affect the self-supervised pretraining performance of SparK, revealing that conventional random masking struggles to jointly model local details and global hierarchical structures. To address this, we propose Structured Mesh Masking: an image is partitioned into multi-scale grids, and tokens are masked in a hierarchical, coarse-to-fine manner—enabling joint optimization of sparsity and hierarchy. We integrate Mesh Masking into the SparK framework, jointly optimizing hierarchical feature reconstruction and contrastive representation learning. On ImageNet-1K linear evaluation, our method achieves a +1.8% top-1 accuracy gain over the baseline. To our knowledge, this is the first work to incorporate explicit grid-based geometric structure into masking design, demonstrating that mask geometry serves as a critical inductive bias for visual representation quality. Our approach establishes a new paradigm for sparse, hierarchical self-supervised learning.

This work clarifies the functional boundaries and necessity of core Transformer components—tokenization, embedding/un-embedding, masking, positional encoding, and padding—addressing widespread conceptual ambiguity in their mechanistic roles. Targeting ML engineers, we propose an incremental, invertibility-based analytical framework: using binary (0/1) sequences as probes, we systematically introduce each component via a “zero-one construction” and empirically validate its irreplaceability in the encode-decode pipeline. Implemented lightweightly in PyTorch, our framework supports manual attention matrix construction, explicit positional embedding injection, and interpretable mask design. Experiments demonstrate significantly improved conceptual accuracy among learners. Notably, we provide the first empirical verification that, in the absence of self-attention, positional encoding combined with padding alone suffices for basic length-aware tasks.

To address the challenges of few-shot learning, sparse labeling, heterogeneous (numerical and categorical) conditioning variables, and constrained computational resources in engineering applications, this paper proposes a masked conditional generative paradigm. We design a unified learnable embedding to jointly model heterogeneous conditions and introduce a masked conditional scheduling mechanism that explicitly simulates missing conditions during training to enhance robustness to incomplete inputs. Furthermore, we construct a lightweight collaborative architecture integrating a variational autoencoder and a latent diffusion model, coupled with knowledge distillation from pre-trained large models. Experiments on 2D point cloud and engineering image datasets demonstrate that the method enables efficient training with only a small number of labeled samples; achieves a 32% reduction in Fréchet Inception Distance (FID); significantly improves conditional fidelity; and simultaneously ensures strong controllability and high generation quality.

This work investigates inter-layer information propagation in Transformer language models, focusing on how features are encoded, routed, and form cross-layer communication channels within low-rank subspaces. We identify and empirically validate the existence of “position-indexed 3D subspaces,” revealing that “contextual item crowding” is the root cause of failure in sequential sensitivity across multiple items. Methodologically, we integrate singular value decomposition (SVD), residual stream subspace analysis, low-rank feature tracking, and intervention experiments on a synthetic task (Laundry List). Crucially, we achieve the first interpretable weight editing and representation intervention grounded in this subspace structure: on the Laundry List task, accuracy improves by over 20%; we successfully predict cross-layer attention interactions; and we provide faithful, mechanistic attributions for model failures.

This work addresses the inefficiency of conventional random masking strategies in language model pretraining, which often fail to identify tokens most valuable for learning. The authors propose a dynamic masking approach that prioritizes tokens with high information content and prediction uncertainty, as measured by the model’s own predictive entropy. Notably, this method introduces a self-masking mechanism that operates without requiring an external reference model. By further integrating knowledge distillation into the training process, the approach significantly enhances both training efficiency and downstream performance. Evaluated on the GLUE benchmark, the proposed method achieves an average performance gain of 5% over the baseline, with the combined use of dynamic masking and knowledge distillation yielding state-of-the-art overall results.

This work investigates the generalization performance and spectral structure of matrix-valued predictors formed by aggregating multiple masks in masked self-supervised learning under high-dimensional settings. Leveraging random matrix theory within an asymptotic framework where sample size and dimension grow proportionally, the study establishes the first high-dimensional theoretical analysis for masked self-supervised learning. The core contributions include deriving an explicit expression for the generalization error, characterizing the spectral properties of the aggregated predictor, revealing a BBP-type phase transition under spiked covariance models, and identifying the precise threshold conditions under which latent signals can be effectively recovered. The analysis further provides theoretical evidence that this approach outperforms classical PCA in certain structured scenarios.

This work addresses the trade-off between event understanding and generalization capability in existing mask prediction–based audio self-supervised learning methods, which often incur high computational costs. To reconcile efficiency and effectiveness, the authors propose a lightweight Dispersion-Weighted Masking (DWM) strategy that leverages the spectral sparsity of audio spectrograms to dynamically adjust the masking distribution, thereby enhancing representation quality. By prioritizing informative yet sparse regions in the time–frequency domain, DWM significantly reduces computational complexity while consistently improving performance across multiple audio event understanding benchmarks. The approach effectively mitigates the longstanding tension between model efficiency and representational power in audio self-supervised learning.

Traditional supervised learning for image classification often discards masked pixels outright, leading to contextual information loss and degradation of fine-grained discriminative features. To address this, we propose a novel “mask-as-knowledge” paradigm that explicitly treats masked regions as semantically rich auxiliary supervision signals—rather than mere occlusions. Our method employs a dual-branch architecture: one branch processes visible pixels, while the other reconstructs masked regions; both branches are jointly optimized via classification loss and mask reconstruction loss, thereby enforcing local–global contextual consistency. This relearning mechanism is architecture-agnostic, seamlessly integrating with both CNNs and Transformers. Extensive experiments on multiple fine-grained visual recognition benchmarks demonstrate significant performance gains, validating the approach’s effectiveness in enhancing feature diversity and preserving discriminative details without requiring architectural modifications.

This work addresses the inefficiency and distributional mismatch in Masked Diffusion Models (MDMs), where training with excessive random masking incurs high computational costs and diverges from the structured masking used during inference. To bridge this gap, the authors propose Progressive Unmasking via Mask Alignment (PUMA), a novel approach that adaptively reshapes the forward masking process to align the training-time mask distribution with that of inference, thereby emphasizing effective masking patterns. PUMA is the first method to achieve consistency between training and inference masking strategies, substantially reducing redundant computation and accelerating convergence while remaining compatible with techniques such as autoregressive initialization. Experiments on a 125M-parameter model demonstrate that PUMA speeds up pretraining by approximately 2.5× without compromising—and in some cases even improving—generation quality.