HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers

2026-06-11

AI Summary

This work proposes HYDRA-X, the first natively unified image-video multimodal foundation model based on a single Vision Transformer architecture that enables joint tokenization. To balance spatiotemporal reconstruction fidelity with semantic awareness, the model incorporates frame-level causal temporal attention and a hierarchical temporal compression mechanism, complemented by a lightweight decompression module that embeds a shared semantic structure within a compact latent space. By relocating editing interactions to the tokenizer’s latent layer and employing joint image-video teacher supervision, HYDRA-X significantly enhances training consistency and convergence speed. Evaluated at a 7B dense parameter scale, the model achieves state-of-the-art performance across both image and video understanding and generation tasks, establishing a critical foundation for unified tokenization in multimodal foundation models.

Abstract

Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space. In this paper, we present HYDRA-X, the first UMM that unifies image and video tokenization within a single Vision Transformer (ViT). Our design is driven by two core challenges: efficiently injecting spatiotemporal reconstruction capability into a native ViT, and embedding image- and video-level semantic awareness into the latent space. To address the first, comprehensive ablations reveal two key findings: (1) frame-level causal temporal attention suffices for visual reconstruction, whereas full spatiotemporal attention degrades it; and (2) hierarchical temporal compression substantially outperforms single-step alternatives. To tackle the second, we propose a lightweight decompressor that upsamples temporally compressed features under joint image-video teacher supervision, thereby enforcing complementary semantic structures within the compact latent space. Building on this holistic tokenizer, we further propose a principled improvement of the editing pipeline: source-target interaction should occur at the latent level inside the tokenizer rather than at the semantic level inside the LLM, substantially improving editing consistency and accelerating convergence. Instantiated at the 7B dense model, HYDRA-X achieves strong performance across image and video understanding and generation tasks, paving the way for future unified-tokenizer UMMs.

Problem

• unified multimodal models
• visual tokenization
• spatiotemporal reconstruction
• semantic awareness
• Vision Transformer

Innovation

• unified multimodal models
• holistic visual tokenizer
• spatiotemporal attention
• temporal compression
• latent-level editing