SKIP: Sparse Keyframe Interpolation Paradigm for Efficient Embodied World Models

2026-05-30

AI Summary

This work addresses the high computational cost and loss of critical events associated with frame-by-frame video generation in embodied world models for long-horizon manipulation tasks. To overcome these limitations, the authors propose SKIP, a novel sparse-to-dense video generation framework that preserves task-relevant events by first identifying and generating only keyframes sparsely, then reconstructing the full sequence using an action-conditioned interpolator and a learned interval predictor. Evaluated on the LIBERO benchmark, SKIP achieves a 4.16× speedup in inference and reduces Fréchet Video Distance (FVD) by 89.0%. Notably, when generated videos fully replace real demonstrations for policy training, task success rates drop by only 1.3 and 6.7 percentage points in simulation and on real robots, respectively, demonstrating that SKIP significantly accelerates generation while maintaining or even enhancing visual fidelity and preservation of task-critical information.

Abstract

Embodied world models have emerged as a promising paradigm in robotics by predicting how robot actions affect the surrounding scene. However, the rollout inference remains computationally expensive in pixel space, as long-horizon manipulation videos typically have to be generated frame by frame. This cost cannot be easily reduced by indiscriminately dropping frames, since downstream policies rely on complete preservation of sparse task-relevant events such as approach, contact, grasp, and release. To address this challenge, we propose Sparse Keyframe Interpolation Paradigm (SKIP), an event-preserving sparse-to-dense framework that avoids dense frame-by-frame generation. SKIP first identifies task-relevant keyframes by leveraging robot-aware multimodal features. It then synthesizes only these keyframes with a sparse video diffusion model. A learned gap predictor and an action-conditioned interpolator subsequently reconstruct the missing intervals according to the robot actions. On LIBERO, SKIP generates dense rollouts $4.16\times$ faster than a dense baseline while improving visual fidelity and reducing aggregate FVD by $89.0\%$. Importantly, SKIP-generated videos are effective policy-training data. Even when they fully replace real demonstrations, $π_{0.5}$ success drops only $1.3$ pp in LIBERO simulation and $6.7$ pp on the real robot, whereas fully dense frame-by-frame generation collapses by $48$ to $58$ pp.

Problem

• embodied world models
• keyframe interpolation
• sparse video generation
• robotic manipulation
• task-relevant events

Innovation

• Sparse Keyframe Interpolation
• Embodied World Models
• Video Diffusion
• Action-Conditioned Interpolation
• Efficient Rollout Generation