MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

2026-07-28

AI Summary

This work proposes a unified, decoder-only architecture for arbitrary-to-arbitrary multimodal modeling that treats all modalities symmetrically, eschewing modality-specific components such as dedicated heads, loss functions, or task pipelines. By employing a single tokenization strategy and end-to-end training, the model fully leverages powerful pretrained decoders without requiring modality-tailored structures or training procedures. This approach achieves, for the first time, fully symmetric arbitrary-to-arbitrary generation, enabling cross-modal chained generation and self-verification within a single framework. Evaluated across multiple benchmarks, the model matches or surpasses specialized and multitask baselines out of the box, demonstrating exceptional zero-shot and general-purpose performance without task-specific adaptation.

Abstract

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.

Problem

• any-to-any modeling
• decoder-only models
• multimodal learning
• pre-trained models
• modality symmetry

Innovation

• decoder-only
• any-to-any modeling
• multimodal learning
• symmetric modality handling
• cross-modal self-verification