What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

2026-07-25

AI Summary

Contrastive vision-language models such as CLIP exhibit insensitivity to negation due to representational collapse in their final layers. This work reveals for the first time that negation signals are preserved in intermediate layers of the text encoder but vanish by the final layer. To address this, the authors propose PeakPatch, a lightweight post-processing framework that recovers negation cues from the syntactically richest “peak layer” without updating CLIP’s frozen parameters, using only 3.5% additional parameters. PeakPatch employs an Embedding and Score Correction Network (ECN/SCN) that leverages cross-attention to predict embedding offsets and score adjustments. On NegBench, the method achieves 74.3% on COCO MCQ (+35.1) and 65.5% on VOC MCQ, substantially outperforming fine-tuning baselines, and demonstrates strong generalization in out-of-distribution negation retrieval and text-to-image generation tasks.

Abstract

Contrastive vision-language models such as CLIP map semantically opposite phrases (e.g., "a dog" vs. "not a dog") to nearly identical embeddings, rendering them insensitive to negation. We attribute this failure to a phenomenon we call Representational Collapse: by tracking compositional divergence and visual alignment across the CLIP text encoder, we show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises, producing a syntax-blind final representation. To recover the lost negation signal without altering pretrained weights, we propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak. An Embedding Correction Network (ECN) uses cross-attention to extract a negation-specific signal from the peak layer, anchored to a stable baseline, and predicts a deviation vector that re-injects the lost syntax into the final-layer embedding space. A complementary Score Correction Network (SCN) predicts bounded scalar score offsets for discriminative tasks. Both modules are trained jointly end-to-end while all CLIP parameters remain frozen, adding only 5.2M parameters (3.5% of the backbone) and preserving the standard cosine similarity interface. On NegBench, PeakPatch achieves 74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoder fine-tuning method) and 65.5% on VOC MCQ, while outperforming all fine-tuning baselines on fully out-of-distribution negation retrieval despite training only 3.5% of the parameters. The corrected embeddings also transfer to text-to-image generation (+18.4 negation score) and generalize across ViT-B/32, ViT-L/14, and SigLIP backbones. Project URL: https://stevencylu.github.io/PeakPatch/.

Problem

• negation
• contrastive vision-language models
• representational collapse
• CLIP
• compositional syntax

Innovation

• Representational Collapse
• Negation Recovery
• Frozen Feature Correction
• Compositional Syntax
• Post-hoc Embedding Adjustment