From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion

2026-06-10

AI Summary

This work addresses the longstanding challenge in multimodal image fusion of simultaneously preserving fine local details and coherent global appearance. To this end, the authors propose a novel architecture that leverages a frozen, pretrained image tokenizer to compress global appearance into a compact 1D token sequence, while maintaining a parallel 2D spatial pathway to reconstruct local structures. A Selective Token Editing (STE) mechanism is further introduced to lightly modulate global consistency without requiring additional loss terms. The proposed method achieves state-of-the-art overall performance across four standard benchmarks, significantly enhancing both global coherence and local fidelity.

Abstract

Multimodal image fusion aims to integrate complementary information from different modalities into a fused image that preserves rich local details while maintaining globally consistent appearance. Existing approaches build shared representations on 2D feature grids, which excel at modeling local structures but offer limited leverage over image-level global appearance factors. To balance these objectives, we introduce a compact 1D token interface based on a frozen pretrained image tokenizer for modeling non-local appearance/base factors. Rather than using the tokenizer as a reconstruction backbone, our design uses the 1D token space as a global carrier while retaining the 2D spatial pathway for local structure restoration. Specifically, we introduce Selective Token Editing (STE), which sparsely updates/replaces a small set of critical tokens, providing a lightweight mechanism to steer global appearance coherence while keeping the fusion backbone unchanged and avoiding extra losses. Experiments on four commonly used benchmarks show that our method achieves the best overall performance, with consistent, multi-metric improvements in both global coherence and local fidelity. Project page: https://zju-xyc.github.io/1D-Fusion-Project-Page/

Problem

• multimodal image fusion
• global appearance consistency
• local detail preservation
• shared representation
• 2D vs 1D representation

Innovation

• 1D token representation
• Selective Token Editing
• multimodal image fusion
• global appearance modeling
• pretrained tokenizer