Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

2026-07-25

AI Summary

This work addresses the "bag-of-concepts" effect in dual-encoder vision-language models, which arises from similarity aggregation mechanisms and undermines compositional reasoning for logically constrained queries (e.g., “umbrella and no person”). To overcome this limitation, the authors propose a factorized reasoning framework that decouples concept evidence extraction from logical constraint enforcement. They introduce Logic Constraint Score Editing (LCSE), a training-free method that explicitly performs logical inference on top of frozen encoders. The study identifies similarity aggregation as the primary cause of compositional failure and introduces FACTOR-Bench, a new evaluation benchmark. Experiments show that LCSE achieves 85.5% accuracy on FACTOR-Bench (90.7% with SigLIP-2), substantially outperforming the best fine-tuned baseline at 73.2%, and improves accuracy from 27.2% to 65.2% on NegBench COCO MCQ while preserving original retrieval performance.

Abstract

Dual-encoder vision-language models (VLMs) expose a similarity interface that enables zero-shot retrieval but fails compositional constraints: queries like "umbrella and no person" retrieve images containing both, even when concept detection is reliable. We trace this to an interface-level Bag-of-Concepts effect, where similarity scores approximate mean pooling of concept evidence regardless of operators. Although operator-dependent signals exist in text embeddings, they are too weak or misaligned to affect rankings. Fine-tuning does not reliably resolve this failure because the dominant bottleneck is how similarity aggregates evidence rather than what encoders represent. We propose factored inference, which separates evidence extraction from constraint execution, and introduce LCSE (Logic-Constrained Score Editing), a training-free method that executes constraints externally using concept scores from frozen encoders. We also introduce FACTOR-Bench, where LCSE achieves 85.5% accuracy versus 73.2% for the best fine-tuned baseline, 90.7% when applied to SigLIP 2, and improves NegBench COCO MCQ accuracy from 27.2% to 65.2% while preserving retrieval performance.

Problem

• compositional reasoning
• vision-language models
• zero-shot retrieval
• logical constraints
• dual-encoder

Innovation

• factored inference
• dual-encoder VLMs
• logic-constrained reasoning
• LCSE
• compositionality