ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression

2026-07-30

AI Summary

This work addresses the limitations of existing learning-based video compression methods, which rely solely on RGB frames for motion estimation and struggle to accurately model temporal redundancy under challenging conditions such as fast motion, blur, occlusion, low texture, poor illumination, and abrupt brightness changes. To overcome this, the paper introduces event camera data into learning-based video compression for the first time, proposing a complementary motion representation (CMR), spatial energy and redundancy calibration (SERIC), and an energy-aware routing (EAR) mechanism. These components jointly refine inter-frame optical flow estimated from RGB inputs while retaining RGB-only encoding, thereby balancing efficiency and performance. Consistent gains are achieved across BS-ERGB, HQ-EVFI, and CED datasets, with up to 20.80% BD-rate savings in PSNR-RGB and 22.14% in MS-SSIM-RGB on BS-ERGB.

Abstract

Learned video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two datasets.

Problem

• learned video compression
• motion estimation
• event cameras
• temporal redundancy
• RGB frames

Innovation

• Event camera
• Learned video compression
• Motion refinement
• Complementary representation
• BD-rate savings