Multiple Hypothesis Flow Estimation for Video Frame Interpolation under Matching Ambiguity

2026-08-07

AI Summary

Traditional single-hypothesis optical flow often leads to ghosting, structural distortion, or blurriness in interpolated frames when confronted with ambiguous matching regions such as repetitive textures, symmetric structures, or motion blur. This work proposes the first multi-hypothesis optical flow estimation framework that operates without ground-truth flow supervision. By maintaining multiple candidate correspondences and employing a reliability-guided routing mechanism to select the optimal hypothesis, the method avoids soft blending and instead refines each hypothesis independently through anchor initialization and local attention. This approach significantly enhances interpolation accuracy, achieving state-of-the-art performance in terms of LPIPS and DISTS metrics on MA-HD and several standard video frame interpolation benchmarks, effectively mitigating ghosting artifacts and structural distortions.

Abstract

Many flow-based video frame interpolation (VFI) methods synthesize an intermediate frame by estimating optical flow fields, warping the two input frames, and blending the warped observations. These latent flow fields are typically learned through image-level reconstruction supervision without direct flow annotations. In ambiguous regions containing repetitive or stochastic textures, rotating symmetric structures, or fast motion with blur, the matching evidence for a single query may contain multiple comparable and spatially separated peaks. Although the ground-truth intermediate frame provides indirect supervision, it may not uniquely identify the latent correspondence in ambiguous regions.When several locations provide multiple plausible matches, a single-flow estimator can retain only one displacement and discard the remaining candidates. If the selected match is incorrect or inconsistent with those of neighboring pixels, warping samples content from mismatched locations, producing ghosting, structural distortion, or blur.To address this limitation, we propose a multiple hypothesis flow estimation framework that preserves top-K candidate correspondences and selects one per location through a reliability-guided router. Each hypothesis is initialized from a coarse matching anchor and refined separately through anchor-centered local attention. Frame synthesis is thus conditioned on one selected flow-appearance hypothesis rather than a soft combination of candidate motions.Experiments on the proposed MA-HD benchmark and public VFI benchmarks show that our method achieves the best LPIPS and DISTS among the compared methods.

Problem

• video frame interpolation
• optical flow
• matching ambiguity
• multiple hypothesis
• correspondence estimation

Innovation

• multiple hypothesis flow estimation
• video frame interpolation
• matching ambiguity
• reliability-guided router
• local attention refinement