ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation

2026-07-27

AI Summary

This work addresses the challenges of unreliable initial prompts and unverified temporal predictions in training-free referring expression video object segmentation by proposing a feedback-driven agent that requires no model training. The method integrates a multimodal large language model with a promptable segmentation model, enabling closed-loop spatial and temporal refinement through mask-guided spatial refinement and a video-level mask reflection mechanism—without updating any model parameters. Specifically, it employs iterative optimization of mask-guided bounding boxes and point prompts, coupled with a re-propagation strategy to enhance consistency across frames. Evaluated on Ref-VPS and ReasonVOS, the approach achieves Q scores of 69.7 and J&F scores of 67.2, respectively, substantially improving the reliability and robustness of training-free video object segmentation.

Abstract

Referring video object segmentation (RVOS) requires segmenting a target specified by natural language throughout a video. Recent agentic approaches combine multimodal large language models with promptable segmentation models to perform RVOS without task-specific training. However, most pipelines rely on one-shot spatial grounding followed by mask propagation, leaving both the initial prompts and temporal predictions largely unverified. We introduce ReflexTrack, a training-free, feedback-driven agent that closes this loop at both spatial and temporal levels. Mask-guided Spatial Refinement evaluates the mask induced by the current keyframe prompt and iteratively updates the bounding box together with positive and negative points, yielding a more reliable initialization. Video-level Mask Reflection assesses the complete mask sequence, localizes unreliable intervals, selects complementary repair keyframes, and generates candidate predictions through mask-guided re-propagation. Only candidates that provide a verified improvement are used to update the affected intervals, preserving reliable predictions elsewhere. All components remain frozen during inference. ReflexTrack achieves an overall $\mathcal{Q}$ score of $69.7$ on Ref-VPS and a $\mathcal{J}\&\mathcal{F}$ score of $67.2$ on ReasonVOS. These results demonstrate that prediction-level feedback substantially improves the reliability of training-free RVOS.

Problem

• Referring Video Object Segmentation
• training-free
• spatial grounding
• temporal prediction
• mask reliability

Innovation

• feedback-driven
• training-free
• referring video object segmentation
• mask refinement
• temporal consistency