masking strategy design

Existing self-supervised learning methods employ random masking, which disregards the intrinsic spatiotemporal (for video) or spectral (for audio) structural properties of multimodal signals, thereby limiting representation quality. To address this, we propose a physics-aware structured noise masking approach: colored noise masks—tailored to each modality’s physical characteristics—are generated via white noise filtering, enabling fully data-free, prior-free, and modality-adaptive mask design. This is the first work to introduce physically interpretable structured noise into masking-based representation learning, preserving the structural constraints of the original signal. Extensive experiments on video and audio self-supervised masked modeling demonstrate that our method significantly outperforms random masking baselines, yielding consistent and substantial gains in downstream task performance—without incurring any additional computational overhead.

This work addresses the challenges of multimodal image fusion under adverse weather conditions, where severe image degradation, feature distortion, and difficulties in modeling cross-modal complementarity hinder performance. To tackle these issues, the authors propose a mask-guided fusion framework that simplifies training through pseudo-ground-truth supervision and introduces a dynamic mask generation mechanism based on the mapping relationship between fused outputs and source images. This mechanism enables the network to selectively focus on informative features during cross-modal interaction. Furthermore, by integrating mask-guided learning with a task-coupled, degradation-aware strategy, the method jointly optimizes feature restoration and fusion. Notably, it incorporates a novel mask-guided cross-modal cross-attention module, achieving consistent and significant improvements over state-of-the-art approaches on both synthetic and real-world datasets in terms of visual quality, quantitative metrics, and downstream task performance.

In supervised learning, strong random masking augmentation often induces training instability. To address this, we propose Masked Sub-model (MaskSub): a dual-model framework where the main model undergoes standard supervised training, while a dedicated sub-model handles masked inputs; a class-wise self-distillation with relaxed loss is introduced to mitigate optimization oscillations. This is the first method to stably enable strong masking augmentation in purely supervised settings—without requiring contrastive objectives or reconstruction targets. Our core innovations are the synergistic dual-model architecture and the relaxed distillation loss, which jointly balance augmentation strength and training stability. Extensive experiments across diverse architectures—including DeiT-III, MAE, CLIP, ResNet, and Swin—and multiple training paradigms demonstrate consistent performance gains, accelerated loss convergence, and superiority over state-of-the-art baselines. The code is publicly available.

This work addresses the computational redundancy in masked diffusion models, which denoise entire images during inference despite large masked regions. To overcome this inefficiency, the authors propose MASQ, a hardware-software co-designed accelerator architecture that integrates staged multi-precision quantization (MXINT8/4/2), timestep-aware scheduling, mask-aware computation, and customized non-matrix operation optimizations. MASQ introduces a novel strategy that jointly leverages spatial semantic importance and dynamic precision allocation, featuring a block-level multi-precision compute engine and a dedicated mask management unit. Experimental results demonstrate that MASQ achieves up to 16.06× speedup and 4.93× higher energy efficiency compared to NVIDIA A100 and Orin NX platforms, while preserving high-fidelity generation quality.

Existing Transformer-based shadow removal methods often incorporate shadow priors through complex modifications to the attention mechanism, resulting in bloated architectures and high computational overhead. This paper proposes a lightweight shadow-aware Vision Transformer (ViT) framework. Its core innovation lies in explicitly embedding the shadow mask into the patch embedding layer—specifically at the *front end*, rather than within the attention modules—enabling efficient integration of shadow priors at the very earliest stage of feature extraction. This design avoids redundant structural alterations to self-attention, relying solely on standard multi-head self-attention and supervised mask guidance. Evaluated on ISTD, ISTD+, and SRD benchmarks, our method surpasses state-of-the-art approaches in both reconstruction accuracy—especially within shadow regions—and inference efficiency, while using significantly fewer parameters. The source code is publicly available.

This work addresses the computational redundancy and spurious background correlations inherent in modern vision models due to dense processing of spatial feature maps, which compromise robustness and interpretability. The authors propose a differentiable energy minimization framework that formulates feature selection as an energy competition mechanism, combining unary importance costs with pairwise spatial consistency penalties. A lightweight energy-based masking layer is integrated into the convolutional backbone to enable input-adaptive dynamic sparsification—without requiring a fixed budget, heuristic scoring functions, or pixel-level supervision. The method preserves classification accuracy while significantly enhancing robustness against structured occlusions, produces semantically focused and highly interpretable masks, and demonstrates superior performance in deletion tests, substantially outperforming baseline approaches such as magnitude-based pruning.

This work addresses the limited generalization of existing camouflaged object detection methods, which often rely on modality-specific architectures or handcrafted fusion strategies. To overcome this, we propose a modality-agnostic multimodal prompting mechanism that enables parameter-efficient fine-tuning of the Segment Anything Model (SAM) to seamlessly integrate arbitrary auxiliary modalities. Our approach unifies data-driven content features with knowledge-driven prompt representations and incorporates a lightweight mask refinement module to enhance boundary accuracy. Extensive experiments across multiple benchmarks—including RGB-Depth, RGB-Thermal, and RGB-Polarization—demonstrate that the proposed method significantly improves detection performance and exhibits strong cross-modal generalization capabilities.

Existing multimodal pretraining approaches typically treat all modalities equally, often failing to adequately optimize representations of the modality most critical for downstream tasks. This work proposes a model-agnostic pretraining strategy that, during masked modeling, enhances the masking difficulty, loss weight, and decoder capacity specifically for the target modality, thereby steering the learning process toward the modality required by downstream applications. Notably, this method introduces explicit modality bias during pretraining without modifying the shared encoder or requiring additional supervision. Evaluated on wireless signal constellation diagram tasks, the approach achieves significant improvements in downstream fine-tuning performance using only existing data and computational resources, demonstrating its effectiveness and practicality.

Multimodal models are highly vulnerable to adversarial attacks due to strong inter-modal dependencies; however, existing defense methods overlook the heterogeneous contributions of individual modalities to overall robustness, resulting in suboptimal defense efficacy. To address this, we propose a vulnerability-aware adversarial training framework. First, we introduce a modality-specific vulnerability quantification mechanism that identifies robustness bottlenecks per modality via first-order approximate adversarial perturbations. Second, we design modality-adaptive regularization terms that enhance robustness without compromising task accuracy. Our approach uncovers a critical blind spot in conventional multimodal adversarial training—its neglect of modality-wise robustness disparities. Extensive experiments on three standard multimodal benchmarks demonstrate consistent and significant improvements in adversarial robustness: +12.73%, +22.21%, and +11.19%, respectively—substantially outperforming state-of-the-art methods.

This work addresses the inefficiency and distributional mismatch in Masked Diffusion Models (MDMs), where training with excessive random masking incurs high computational costs and diverges from the structured masking used during inference. To bridge this gap, the authors propose Progressive Unmasking via Mask Alignment (PUMA), a novel approach that adaptively reshapes the forward masking process to align the training-time mask distribution with that of inference, thereby emphasizing effective masking patterns. PUMA is the first method to achieve consistency between training and inference masking strategies, substantially reducing redundant computation and accelerating convergence while remaining compatible with techniques such as autoregressive initialization. Experiments on a 125M-parameter model demonstrate that PUMA speeds up pretraining by approximately 2.5× without compromising—and in some cases even improving—generation quality.