Problem
Research questions and friction points this paper is trying to address.
Neural Video Compression
Event-guided
Cross-modal
Compression Efficiency
Temporal Context
Event-guided Neural Video Compression
🤖 AI Summary
This study addresses the limited compression efficiency of existing neural video codecs that rely solely on RGB frames, which results in insufficient exploitation of temporal context. To this end, this work proposes ENVC, the first event-guided neural video coding framework. By incorporating event streams to assist RGB encoding, the model achieves cross-modal complementary gains through an event-guided motion prior and a conditional predictor. Furthermore, a synthetic paired-data training strategy is devised, combined with multi-scale feature fusion and a gated temporal context refinement mechanism to enhance performance. Extensive evaluations across six benchmarks demonstrate that ENVC achieves average bitrate reductions of 39.13% in PSNR and 67.63% in LPIPS compared to DCMVC, significantly improving compression efficiency.
📝 Abstract
Neural video codecs derive motion and temporal contexts mainly from RGB frames, leaving room for cross-modal guidance from complementary temporal observations. Event streams can provide such observations by recording brightness changes between frames. In this work, we propose an Event-guided Neural Video Codec (ENVC) that uses events shared by the encoder and decoder to improve RGB compression efficiency. For motion coding, ENVC forms an event-guided motion prior and codes the remaining motion residual. For frame coding, an event-conditioned predictor supplies multi-scale features for gated temporal context refinement. To support training and evaluation on standard video datasets, we synthesize paired RGB-event data and assess its predictive utility through comparisons with real events. Across six benchmarks, ENVC achieves average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC. Further analyses show that our gains persist on large-motion sequences and that ENVC effectively learns to integrate event information. These results demonstrate the potential of events as a complementary modality for reducing the RGB coding rate. Our model and code are available at https://github.com/kjungwoo03/ENVC.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Event-guided Neural Video Compression
Cross-modal Guidance
Motion Prior
Gated Temporal Context Refinement
Synthetic RGB-Event Data
🔎 Similar Papers
No similar papers found.
