multi-frame semantic fusion

Existing video multimodal fusion methods naively adapt static image fusion techniques, neglecting temporal dependencies and resulting in inter-frame inconsistency. To address this, we propose the first temporal modeling and vision–semantics co-learning framework specifically designed for video fusion. Our method introduces a vision–semantics interaction module and a temporal coordination module, implemented via dual distillation branches—DINOv2 for semantic representation and VGG19 for low-level visual features. We further incorporate a temporal enhancement mechanism, a temporal consistency loss, and a dedicated evaluation metric suite. Extensive experiments demonstrate that our approach significantly improves weak-information recovery and dynamic coherence, achieving state-of-the-art performance across multiple public video datasets. It attains superior results in all three critical dimensions: visual fidelity, semantic accuracy, and temporal continuity. The source code is publicly available.

RGB-D multimodal fusion mechanisms have long suffered from poor interpretability, and the fundamental nature of cross-modal complementarity remains unclear. Method: This paper establishes the first interpretability analysis framework specifically targeting the fusion process, introducing a joint metric of semantic variance and feature similarity to systematically characterize cross-modal representation consistency, intra-modal evolutionary patterns, and collaborative optimization logic. Through cross-layer feature comparison and quantitative semantic analysis, we identify a prevalent imbalance between consistency and specificity in mainstream fusion strategies. Contribution/Results: We formalize a “specificity-driven inference under consistency constraints” principle that explicates cross-modal complementarity. Our framework provides both theoretical foundations and a verifiable evaluation paradigm for designing trustworthy, generalizable multimodal fusion models.

This work addresses the limitation of existing cross-modal alignment methods, which often conflate semantic and non-semantic information, leading to insufficient semantic consistency and alignment bias caused by modality gaps. To overcome this, we propose a semantic alignment framework based on constrained disentanglement and distribution sampling. Specifically, a dual-path UNet architecture adaptively disentangles visual and linguistic representations into semantic and modality-specific components, aligning only the extracted semantic factors. Furthermore, a multi-constraint optimization strategy combined with distribution-aware sampling is introduced to effectively bridge inter-modality discrepancies, thereby enhancing the reasonableness and robustness of alignment. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches across multiple benchmarks and backbone architectures, achieving performance gains of 6.6% to 14.2%.

Existing unified multimodal models are typically evaluated in isolation on either visual understanding or generation tasks, lacking assessment of semantic consistency across tasks. To address this gap, this work proposes XTC-Bench, a novel evaluation framework that introduces the Continuous Cross-Task Agreement (CCTA) metric to disentangle a model’s internal consistency from its individual task performance at the atomic fact level. The framework leverages structured scene graphs to align comprehension queries with generation prompts and establishes the first reproducible, model-agnostic benchmark for cross-task consistency through fine-grained matching of objects, attributes, and relationships. Experiments across nine state-of-the-art models reveal that high task accuracy does not guarantee strong cross-task consistency, which is primarily influenced by the degree of coupling in cross-modal learning objectives.

This study investigates the semantic alignment mechanism between vision and language deep models under unsupervised conditions. To this end, we conduct deep representation analysis, cross-modal similarity modeling, Pick-a-Pic forced-choice evaluation, and multi-caption/image matching assessment. Results show that semantic alignment peaks at middle-to-late network layers, exhibiting strong semantic sensitivity and robustness to visual appearance variations. Moreover, averaging representations across multiple instances significantly enhances alignment strength—surpassing conventional one-to-one pairing paradigms and better reflecting human fine-grained preferences in many-to-many image-text scenarios. Key contributions include: (1) the first empirical confirmation that unimodal models encode a shared semantic structure consistent with human judgments; and (2) the discovery that aggregating multiple examples improves alignment quality, with substantial gains achieved while preserving semantic fidelity.

This work investigates whether the understanding and generation branches of unified multimodal models share a transferable semantic space. To this end, the authors propose a cross-branch semantic guidance framework that extracts intervention-based semantic directions from the understanding branch and transfers them to the generation branch for controllable image synthesis. Their experiments reveal, for the first time, a semantic asymmetry between the two branches: the understanding branch encodes object-level semantics, whereas the generation branch relies more heavily on low-level appearance features. The proposed method enables effective semantic transfer from understanding to generation, significantly improving the semantic fidelity of generated images; however, reverse transfer yields limited gains, demonstrating that architectural unification does not inherently guarantee semantic alignment.

研究通过3D融合解决无人机开放词汇分割模型的时间语义稳定性问题，提出评估框架分析语义一致性及不确定性。

This study addresses the modality mismatch between non-invasive neural signals and semantic representations, which limits the performance of continuous language reconstruction from brain activity. The authors propose a multi-feature fusion framework that systematically compares linear concatenation and nonlinear cross-attention strategies for the first time, introducing an interactive gating mechanism to jointly integrate static word embeddings (Word2Vec) with dynamic contextual representations (GPT). Experimental results demonstrate that nonlinear fusion based on multi-head cross-attention significantly outperforms alternative approaches, following the hierarchy Cross-Att > Concat > GPT > Word2Vec. These findings highlight the critical role of token-level attributes and context-aware modulation in neural decoding, transcending the limitations of single-feature representations and achieving state-of-the-art performance in non-invasive brain-to-text reconstruction.

This work addresses the lack of principled understanding in existing literature regarding the choice between cross-attention and feature concatenation strategies for multimodal fusion, which has largely relied on empirical heuristics. Through controlled experiments and theoretical analysis, we demonstrate for the first time that feature alignment quality is the key determinant of fusion strategy performance: under pre-aligned features, concatenation consistently outperforms cross-attention by 4.1–5.1 percentage points across all data scales, with its advantage becoming more pronounced as alignment degrades. Building on this insight, we develop a theoretical decision framework grounded in sample complexity and validate our findings using features extracted from ResNet-18 and CLIP ViT-B/32 on controlled datasets.

This work addresses the degradation of cross-modal alignment in vision-language models under out-of-distribution (OOD) scenarios by proposing SynerNet, a novel framework that integrates four synergistic computational units—visual perception, linguistic context, named embeddings, and global coordination—to establish a structured message-passing mechanism for mitigating modality discrepancies. SynerNet innovatively combines multi-agent latent-space naming, semantic context exchange algorithms, and an adaptive dynamic balancing mechanism to enable collaborative optimization of cross-modal semantics. Evaluated on the VISTA-Beyond benchmark, the proposed method achieves accuracy improvements of 1.2% to 5.4% over existing approaches in both few-shot and zero-shot OOD settings, demonstrating its superior robustness and generalization capability.