VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

2026-06-04

AI Summary

Existing video object segmentation methods struggle to actively acquire visual evidence in complex long-form videos, limiting their ability to support tasks requiring fine-grained temporal, spatial, and linguistic reasoning. This work proposes the first multi-round reinforcement learning framework tailored for Reasoning-based Video Object Segmentation (RVOS), which emulates the human cognitive process of progressing from coarse to fine understanding through an iterative spatiotemporal chain-of-thought to localize critical clips and frames. Key innovations include a SEG-aware logit calibration mechanism, decoupled modeling of spatiotemporal-linguistic reasoning trajectories, and VTS-CoT—the first cold-start dataset annotated with complete reasoning trajectories. Experiments demonstrate that the proposed approach significantly enhances segmentation accuracy and robustness in complex scenarios, enabling the model to actively explore and precisely localize targets even without initial cues.

Abstract

Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose \textbf{VideoSEG-O3}, the first multi-turn reinforcement learning framework for RVOS that emulates the human \textit{``coarse-to-fine''} cognitive process. It employs a \textit{multi-turn temporal-spatial chain-of-thought} to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of \texttt{[SEG]} during the RL stage, we introduce \textit{SEG-aware logit calibration}, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a \textit{decoupled thinking trace} to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct \textbf{VTS-CoT}, a specialized cold-start dataset featuring comprehensive reasoning trajectories. The code and models will be released at https://github.com/Dmmm1997/VideoSEG-O3.

Problem

• Reasoning Video Object Segmentation
• temporal dynamics
• spatial details
• linguistic reasoning
• complex references

Innovation

• multi-turn reinforcement learning
• temporal-spatial chain-of-thought
• SEG-aware logit calibration
• decoupled thinking trace
• reasoning video object segmentation