HarmoView: Harmonizing Multi-View Constraints for Identity-Consistent Video Generation

2026-06-09

AI Summary

Existing identity-consistent video generation methods struggle to preserve appearance fidelity under large viewpoint variations and lack effective mechanisms for multi-view input as well as large-scale training data. To address these limitations, this work proposes HarmoView, a novel framework that anchors identity through multi-level ViT feature injection, unifies heterogeneous reference layouts via learnable proxy tokens, and isolates identity features using Jump-RoPE positional encoding. Additionally, a progressive viewpoint curriculum training strategy is introduced. The study also constructs the first large-scale multi-view facial video dataset. Evaluated on a benchmark of 100 curated cases spanning 52 identities, HarmoView significantly outperforms open-source baselines and achieves state-of-the-art performance comparable to leading closed-source systems.

Abstract

Current identity-consistent video generation methods struggle to preserve appearance fidelity under large viewpoint changes. While introducing multi-view reference input offers a natural solution, progress remains constrained by the lack of effective frameworks for multi-view inputs and the scarcity of multi-view data. We address these challenges by proposing HarmoView, a robust framework for identity-consistent video generation that effectively integrates multi-view cues through three architectural refinements complemented by a staged training curriculum. Specifically, we first introduce Multi-level Feature Injection to anchor identity fidelity; by injecting raw ViT features from frontal references alongside text tokens via cross-attention, MFI provides persistent low-level appearance anchors that complement the high-level identity features within DiT blocks, leading to enhanced identity preservation. Then, we employ learnable proxy tokens to unify heterogeneous reference layouts across single-/multi-view settings while simultaneously resolving the reference-view mismatch problem. Jump-RoPE is further developed for identity-wise feature isolation to reduce identity crosstalk. To activate these structural capabilities while preserving the original generative priors, we propose the Progressive View Curriculum. This four-stage training strategy employs view dropout to facilitate a stable transition from vanilla T2V generation to high-fidelity, identity-persistent spatial reasoning. Furthermore, we construct a large-scale multi-view dataset to address the issue of data scarcity. Extensive evaluation on our multi-view benchmark, comprising 100 manually-curated cases spanning 52 unique identities, demonstrates that HarmoView significantly outperforms open-source baselines and matches leading closed-source engines, achieving state-of-the-art performance in identity-consistent video generation.

Problem

• identity-consistent video generation
• multi-view constraints
• appearance fidelity
• viewpoint changes
• multi-view data scarcity

Innovation

• Multi-view video generation
• Identity consistency
• Feature injection
• Progressive training curriculum
• Jump-RoPE