Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback

2026-09-30

AI Summary

This study addresses the insufficient inter-modal information interaction in multimodal representation learning by proposing the MEQ architecture. This method introduces a continuous bidirectional reciprocal feedback mechanism to replace conventional feature concatenation, coupling inputs from different modalities into mutually representative embeddings and yielding their interactive fixed point as the final output. Furthermore, theoretical analysis is provided to guide network design, effectively preventing failure modes. Experimental results demonstrate that MEQ surpasses or matches existing baseline methods on classification and visual grounding tasks, achieving particularly significant improvements in localization accuracy under cross-modal complementary scenarios.

Abstract

This work proposes a mutual feedback architecture, MEQ, that refines the two inputs, of possibly different modalities, into a pair of coupled embeddings such that each embedding reflects the information of the other. The core idea is to incorporate continuous interchange of information between the two inputs. This idea leads to a mutual feedback architecture consisting of two components whose outputs are fed back into the other. The final output of this model is defined as the fixed point of this interaction. We provide theoretical analysis that offers interpretation of this model as well as design choices to prevent failure cases. We show the benefits of MEQ through classification and visual grounding tasks spanning various datasets. Quantitatively, our model outperforms or shows competitive performance on concatenation-based multimodal classification problems. Qualitatively, the proposed interactive mechanism allows the model to progressively refine the visual grounding when paired with complementary modality, thus demonstrating the power of mutual feedback under such settings.

Problem

• Multimodal Representation Learning
• Mutual Feedback
• Visual Grounding
• Coupled Embeddings
• Multimodal Classification

Innovation

• Mutual Feedback
• Multimodal Representation Learning
• Fixed Point
• Visual Grounding
• Reciprocal Interaction