Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions

2026-08-06

AI Summary

This work addresses the challenge that existing methods struggle to simultaneously localize both the content and position of changes when analyzing compositional reasoning with frozen vision encoders, often leading to the “operation laundering” problem. To resolve this, the authors propose a Support-Operation Factorization (SO-OPF) readout mechanism, which integrates a leave-one-unit-out evaluation protocol with an injection alignment design. This approach decomposes unit energy into support saliency and competitive operation posteriors, thereby disentangling compositional binding recovery from grid assignment learning. Evaluated on frozen DINOv3/SigLIP2 features, the method achieves injection accuracies of 0.874 on Shapes3D-Extended and 0.799 on COCO. Under matching supervision, the factorized architecture improves assignment accuracy from 0.653 to 0.841 and effectively eliminates the operation laundering gap.

Abstract

Compositional analysis of frozen vision encoders should determine both what changed and where it changed. Standard factor probes score these axes separately, however, and can reward multiple operations that reuse the same predicted slot. We call this failure operation laundering. We introduce an injectively aligned leave-one-cell-out protocol over support x operation grids and SO-OPF, a readout that factors cell energy into support salience and a competitive operation posterior. This formulation separates two questions that aggregate scores conflate: whether the carrier composes held-out bindings when the grid is known, and whether that grid can be recovered from flat cell labels. With frozen DINOv3 features, known factorial assignment reaches 0.874 injective accuracy on Shapes3D-Extended and 0.799 on globally image-disjoint COCO; learning the assignment from flat labels reaches 0.769 and 0.762, respectively. Under matched-axis-aware supervision on Shapes3D, the factored carrier improves learned-assignment accuracy from 0.653 to 0.841 over a dense carrier and eliminates its laundering gap. SigLIP2 replicates the COCO separation. A rebuilt MuJoCo substrate exposes a boundary: learned-assignment accuracy is 0.569 with DINOv3 and 0.484 with SigLIP2, with substantial slot collapse. Thus factored readout and injective evaluation recover held-out bindings on two substrates while exposing, rather than hiding, a renderer-specific failure boundary; they do not establish universal recovery from flat labels.

Problem

• compositional analysis
• frozen vision encoders
• operation laundering
• injective evaluation
• held-out bindings

Innovation

• Support Operation Factorization
• injective evaluation
• operation laundering
• compositional readout
• frozen vision encoders