🤖 AI Summary
This work addresses the challenge that current vision-language models struggle to emulate humans’ ability to develop shared referential expressions through repeated interaction—a phenomenon known as lexical coordination. To bridge this gap, the authors propose a dynamic semantic framework that explicitly maintains three sets of reference-object binding states and integrates a lightweight perceptual alignment module leveraging SIFT homography, Universal Quality Index (UQI), and image augmentation techniques. This architecture externalizes the lexical coordination process into an inspectable symbolic layer, yielding a transparent and auditable mechanism for referential understanding amenable to fine-grained ablation studies. Evaluated on the Stanford Referring Expression Repetition Game corpus, the model achieves 83.56% accuracy in identifying the target within the top-5 candidates using only a single utterance, and demonstrates robust performance even under conservative evaluation conditions that exclude salient distractors.
📝 Abstract
Humans converge on shared names for novel, hard-to-describe objects through repeated interaction, a process psycholinguists call lexical entrainment. Leading vision-language models fail at this: recent empirical work documents that they do not shorten references, reuse successful expressions, or maintain stable pact state across turns. We present a framework that addresses the gap by externalizing pact state into three explicit, inspectable sets of referent-object bindings ($Γ, Ξ, Ω$), updated by a dynamic-semantics context-change rule. The symbolic layer sits on top of a lightweight perceptual-alignment pipeline that grounds noisy human referring expressions in crowd-sourced imagery via SIFT homographies and the Universal Quality Index. Evaluated on the Stanford Repeated Reference Game corpus (over 15{,}000 director-matcher utterances on abstract tangram stimuli), the framework places the correct target in its top-5 hypothesis set 83.56% of the time from a single director utterance. Human matcher top-1 accuracy on the same corpus is approximately 77-80%. We also report results on a held-out condition in which obvious tangram-adjacent images are excluded from the retrieved set, which provides a more conservative measurement of the grounding signal. Ablations isolate the contribution of each component: SIFT alignment, UQI, query preprocessing, and image augmentation. The central contribution is the combination: a transparent, auditable symbolic layer that recovers the structure of lexical entrainment turn by turn, paired with a perceptual channel whose behavior can be examined ablation by ablation. We also discuss in detail what the framework does not do. It is not interactive, it does not close the loop with the director, and its retrieval-driven perceptual channel is vulnerable to a class of leakage effects that we quantify and bound rather than wave away.