EgoPlay: Event-Triggered Video Editing for Egocentric Streams

2026-07-27

AI Summary

This work addresses the challenge of precise event-triggered editing in egocentric videos, which requires accurate event detection, preservation of pre-event content, and selective editing of subsequent segments. To this end, we propose the first end-to-end bidirectional diffusion editor that jointly models event detection and video editing. Built upon a pretrained video-to-video diffusion Transformer, our method integrates bidirectional and causal architectures to support positive, negative, and multi-event textual prompts, while enabling streaming chunk-wise inference. We introduce a large-scale event-conditioned video-prompt dataset comprising 106K samples and devise an event-aware evaluation protocol. On the Ego4D benchmark, our approach outperforms the state-of-the-art method EgoEdit by 17.7%, 16.9%, and 16.4% in editing quality, visual fidelity, and background consistency, respectively, while using less than half the GPU memory.

Abstract

We introduce EgoPlay, an event-triggered video-to-video editor for egocentric streams, obtained by fine-tuning a pretrained V2V diffusion transformer on event-conditioned data built primarily from Ego4D. Given a monocular video and an event-triggered prompt of the form "when X happens, do Y," EgoPlay infers whether and when event X occurs, preserves pre-event frames, and applies edit Y only to the post-event continuation. Rather than cascading a separate event detector with an editor, EgoPlay learns event recognition, temporal restraint, and pixel-level editing jointly in a single end-to-end model, while also handling negative and multi-event prompts. To support this, we construct a large-scale dataset of 106K event-triggered clip-prompt pairs spanning positive triggers, fabricated-trigger negatives, and multi-event prompts. We then train a bidirectional video diffusion editor with event-triggered supervision and derive a causal variant for chunk-by-chunk streamable inference. We further introduce an event-aware evaluation protocol that separately measures post-trigger editing quality, pre-trigger preservation, and false-trigger robustness. On the Ego4D benchmark, EgoPlay substantially outperforms EgoEdit, the state-of-the-art instruction-based egocentric video editing baseline, with relative gains of 17.7%, 16.9%, and 16.4% in editing quality, visual quality, and background consistency. It also surpasses a VLM-guided detector-editor baseline by 15.7%, 14.5%, and 13.5% on the same metrics, while using less than half the GPU memory.

Problem

• egocentric video editing
• event-triggered editing
• video-to-video generation
• temporal event detection
• instruction-based video editing

Innovation

• event-triggered video editing
• egocentric video
• diffusion transformer
• end-to-end joint learning
• streamable inference