Invisible Shortcuts: Why Vision Encoders Know Your Camera

2026-08-05

AI Summary

This work addresses the tendency of vision models to learn spurious shortcuts during pretraining by implicitly associating semantic labels with invisible metadata—such as device-specific or processing artifacts—leading to performance degradation under distribution shifts. The study presents the first systematic investigation of this issue, employing controlled experiments to quantify the extent to which metadata induces shortcut learning. To mitigate this problem, the authors introduce both training-time and post-training debiasing strategies that are broadly applicable across model architectures. Their approach effectively reduces model reliance on both known and unknown metadata sources, significantly improving out-of-distribution generalization without compromising downstream task performance, while also enhancing robustness in generated image detection.

Abstract

Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces

Problem

• shortcut learning
• metadata traces
• vision encoders
• distribution shift
• generalization

Innovation

• shortcut learning
• metadata traces
• vision encoders
• distribution shift
• out-of-distribution generalization