SCVIB: Editable State-Conditioned Visual Instance Binding forMulti-Turn Personalized Localization

2026-08-14

AI Summary

This study addresses the challenges of state-dependent parsing and insufficient visual evidence utilization in multi-turn personalized grounding. We propose an editable state-conditional visual instance binding task alongside the tt-vg framework. This approach pioneers compiling state events into versioned target state transition trees and introduces a vega mechanism to facilitate conditional grounding with adaptive visual evidence integration. Evaluated on our newly constructed scvib benchmark, the method achieves a joint@0.5 accuracy of 70.27%, surpassing the strongest baseline by 16.20 percentage points. Notably, it demonstrates significant improvements in anti-recency and state rollback scenarios, effectively resolving instance localization difficulties under non-latest states. These results validate the framework’s capability to handle complex temporal state dependencies and enhance robustness in dynamic visual grounding tasks where current context does not reflect the most recent interactions.

Abstract

We introduce editable state-conditioned visual instance binding, a multi-turn localization setting in which several support-defined instances are introduced across turns and protocol-defined state events determine the final target. We instantiate this setting as SCVIB, comprising 1,050 manually verified support--query base pairs and 1,500 episodes spanning five visual domains, three difficulty levels, and four target-state dependency groups. Direct Seq-free inference reaches only 60.13\% Joint@0.5, indicating that resolving the final reference does not ensure effective use of the corresponding visual evidence for query-side localization. We address this gap with TT-VG (Transition-Tree Visual Grounding), which combines a Target-State Transition Tree (TSTT) with Visual Evidence Grounding Adaptation (VEGA). TSTT compiles the visible interaction into protocol-defined events, executes them over versioned target states, and resolves the final-query reference to the corresponding support evidence. Adapted on trajectory-derived same-instance pairs, VEGA performs support-conditioned grounding of the resolved instance using a Visual Evidence Package. TT-VG reaches 70.27\% Joint@0.5; under matched target resolution, VEGA exceeds the strongest comparison method by 16.20 points. Gains over direct inference are largest on Counter-Recency and Rollback, which require routing to non-latest or restored support evidence. Together, these results establish SCVIB as a controlled testbed and highlight the effective use of resolved support evidence for query-side same-instance localization as a central challenge in multi-turn personalized localization.

Problem

• Multi-Turn Personalized Localization
• Visual Instance Binding
• State-Conditioned Grounding
• Support Evidence Utilization

Innovation

• Transition-Tree Visual Grounding
• Target-State Transition Tree
• Visual Evidence Grounding Adaptation
• Editable State-Conditioned Visual Instance Binding
• Multi-Turn Personalized Localization