Problem
Research questions and friction points this paper is trying to address.
omnimodal embeddings
shared parameter space
modality-specific representations
cross-modal collaboration
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
🤖 AI Summary
This study addresses the challenge of disentangling general and modality-specific representations within the shared parameter space of omni-modal embeddings. To this end, we propose a unified framework based on structured adaptation. The core innovation lies in the novel Orthogonal Modal Expert LoRA (OME-LoRA) decomposition pathway, which is coupled with a progressive collaborative routing mechanism. This design establishes modality priors while enabling controlled cross-modal collaboration, effectively balancing modality specialization with interactive fusion. Extensive experiments demonstrate that the proposed method significantly outperforms existing baselines across 81 multimodal tasks encompassing diverse modalities such as images and videos. These results thoroughly validate the effectiveness and generalization capability of our approach for multimodal deep learning.
📝 Abstract
Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Omnimodal Embeddings
Orthogonal Modality-Expert LoRA
Progressive Synergy Routing
Structured Specialization
Cross-modal Collaboration
🔎 Similar Papers
No similar papers found.
