RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning

2026-07-31

AI Summary

This work addresses the sensitivity of visual imitation learning policies to camera viewpoint changes, which stems from the lack of explicit geometric information in RGB images and limits generalization. To mitigate this issue, the authors propose RayViT, a lightweight architecture that injects camera geometry priors into a Vision Transformer (ViT) backbone. Specifically, RayViT replaces the class token with dense ray positional embeddings derived from Plücker ray maps and fuses geometric and visual features via a gated cross-attention mechanism. Additionally, a cosine similarity–based auxiliary loss is introduced to encourage alignment between representations. Evaluated on the RoboCasa multi-task simulation benchmark, RayViT improves policy robustness under viewpoint perturbations by approximately 13 percentage points and enables real-world robots to complete an average of 1.78 additional task stages compared to baseline methods.

Abstract

Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose \textbf{Ray-conditioned Vision Transformer Encoder (RayViT)}, a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Plücker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.

Problem

• visual imitation learning
• camera perturbations
• viewpoint robustness
• geometric cues
• visuomotor skills

Innovation

• Ray-conditioned representation
• Viewpoint-robust imitation learning
• Plücker ray map
• Gated cross-attention
• Geometry-aware visual transformer