When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE

2026-05-29

AI Summary

This work addresses a fundamental mismatch in contrastive learning: the softmax assumption underlying InfoNCE loss under normalized embeddings fails to accurately capture the extreme-value distribution of hard negative samples, thereby limiting performance. To bridge this gap, the paper introduces extreme value theory into contrastive objective design and proposes WEINCE, a parameter-free method that performs endpoint tail correction on logits using anchor-level online batch statistics to better model the tail behavior of hard negatives. Requiring no additional trainable parameters, WEINCE consistently improves frozen-feature evaluation performance across five visual benchmarks, demonstrating the efficacy of statistically grounded correction in contrastive representation learning.

Abstract

InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected. Using extreme value theory, we show that this assumption is often misaligned with the normalized embedding setting used in modern contrastive learning. Motivated by this mismatch, we propose \textsc{WEINCE}, a simple modification of InfoNCE that uses anchor-wise online batch statistics to blend the usual softmax logits with an endpoint shortfall correction, adding no trainable parameters. Across five vision benchmarks, \textsc{WEINCE} yields consistent improvements in frozen-feature evaluation. These results show that a more faithful statistical treatment of hard negatives can improve contrastive objectives.

Problem

• InfoNCE
• softmax
• extreme value theory
• contrastive learning
• hard negatives

Innovation

• Extreme Value Theory
• InfoNCE
• Contrastive Learning
• Hard Negatives
• WEINCE