MV-Actor: Aligning Multi-View Semantics and Spatial Awareness for Bimanual Manipulation

2026-06-09

AI Summary

This work addresses the challenge that existing dual-arm manipulation methods struggle to effectively share multi-view semantic information and establish reliable 3D spatial perception in complex or noisy environments. To overcome this limitation, the paper proposes a unified semantic-spatial representation framework that, for the first time, enables deep cross-view semantic sharing and alignment of spatial perception. Key innovations include multi-view semantic interaction, a semantic-spatial token coupling mechanism, feedforward reconstruction-guided depth refinement, and optimization of reliable metric anchors tailored for consumer-grade RGB-D sensors. The proposed method achieves an average success rate of 87.8% on the PerAct2 simulation benchmark and significantly outperforms current RGB and RGB-D baselines in real-world scenarios.

Abstract

Robotic manipulation has been widely applied in industrial scenarios. Compared with single-arm manipulation, bimanual manipulation is equipped with multiple cameras to capture information from different viewpoints. However, existing multi-view policies encode each view independently or fuse view features shallowly, resulting in limited sharing semantic perception and unreliable spatial awareness. In this paper, we propose \textbf{MV-Actor}, a multi-view perception framework that builds a unified semantic-spatial representation for bimanual manipulation. First, MV-Actor performs Multi-view Semantic Interaction to share semantic perception across views. Then it uses Semantic-Spatial Token Interaction to ground visual semantics with feed-forward reconstruction model features and acquire reliable spatial awareness. Finally, a Guided Metric Depth Repair module refines degraded sensor depth to provide more reliable metric anchors under consumer-grade depth noise. In simulation experiments conducted on the PerAct2 bimanual benchmark, MV-Actor achieves a state-of-the-art average success rate of 87.8\%. In real-world evaluations with more frequent viewpoint changes and unstable consumer-grade depth, MV-Actor outperforms both RGB and RGB-D baselines, further demonstrating the benefit of sharing semantic perception and reliable spatial awareness for bimanual manipulation.

Problem

• bimanual manipulation
• multi-view perception
• semantic perception
• spatial awareness
• depth sensing

Innovation

• multi-view perception
• semantic-spatial representation
• bimanual manipulation
• depth refinement
• cross-view interaction