Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training

2026-07-22

AI Summary

This work addresses the challenge in visible-infrared (VIS-IR) pre-training that imaging modality discrepancies render certain spatially aligned regions unreliable, thereby limiting representation learning and downstream transfer performance. To mitigate this, the authors propose Importance-Aware Sampling (IAS), the first approach to incorporate sampling importance into VIS-IR pre-training. IAS dynamically computes block-level weights using an infrared structural prior, employs a lightweight sampler to learn a soft importance mask, and introduces a block-level curriculum learning strategy that progressively expands from highly reliable to more challenging regions. Designed as a plug-and-play module, IAS is compatible with both patch/correlation alignment and image-level contrastive learning frameworks such as UNIV and ImageBind. Extensive experiments demonstrate consistent and significant improvements over strong baselines across multiple VIS-IR benchmarks, including infrared semantic segmentation, object detection, visible-light semantic segmentation, and cross-modal retrieval tasks.

Abstract

Visible-infrared (VIS-IR) alignment is a key pre-training task for robust multi-sensor perception. Most existing methods use uniform patch-wise contrastive learning, but this can be unreliable in VIS-IR data because imaging-physics differences make some spatially paired regions inherently less comparable, and aligning them with equal strength hinders representation learning and downstream transfer. In this paper, we revisit VIS-IR pre-training from a sampling perspective and propose Importance-Aware Sampling (IAS), which adjusts training emphasis based on patch reliability. Specifically, IAS (i) derives patch weights from infrared structural cues and uses them to reweight the contrastive objective; (ii) learns a soft importance mask with a lightweight sampler, optionally warm-started from the hand-crafted prior; and (iii) employs a patch curriculum learning strategy that gradually expands from high-reliability regions to harder patches. It is worth noting that IAS is plug-and-play and works with both patch-/correlation-level alignment (e.g., UNIV-style) and image-level contrastive baselines (e.g., ImageBind-style). Extensive experiments on multiple VIS-IR benchmarks demonstrate consistent improvements over strong baselines, including for IR semantic segmentation, IR object detection and VIS semantic segmentation and cross-modal retrieval task. Code will be released on https://github.com/KlayMa527/IAS.

Problem

• Visible-infrared alignment
• Patch sampling
• Contrastive learning
• Multi-sensor perception
• Representation learning

Innovation

• Importance-Aware Sampling
• Visible-Infrared Pre-Training
• Patch Reliability
• Curriculum Learning
• Contrastive Learning