EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE

2026-07-30

AI Summary

This work addresses the high cost and limited diversity of real-world egocentric video data by proposing an egocentric world-action simulator based on a pretrained video generation model. The method introduces Online Anchored Projection Memory (OAPM) to preserve the initial 3D scene anchor and dynamically update scene states, and designs Action-3D Rotary Position Encoding (A3D-RoPE) to inject end-effector motion—represented in camera-perceived 3D coordinates—into cross-attention for precise and controllable action synthesis. Experiments demonstrate that the approach significantly enhances the generalization of downstream world-action models, improving out-of-distribution success rates on real robots from 77% to 84% in single-arm tasks and from 53% to 70% in dual-arm tasks.

Abstract

Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present \method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. \method{} builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 \method-generated trajectories improves out-of-distribution real-robot success from 77\% to 84\% on single-arm tasks and from 53\% to 70\% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization.

Problem

• egocentric video
• data scarcity
• embodied AI
• manipulation data
• real-world generalization

Innovation

• EgoGenesis
• Online Anchored Projective Memory
• Action-3D RoPE
• egocentric video synthesis
• geometry-aware conditioning