multi-source feature integration

This work addresses the lack of principled understanding in existing literature regarding the choice between cross-attention and feature concatenation strategies for multimodal fusion, which has largely relied on empirical heuristics. Through controlled experiments and theoretical analysis, we demonstrate for the first time that feature alignment quality is the key determinant of fusion strategy performance: under pre-aligned features, concatenation consistently outperforms cross-attention by 4.1–5.1 percentage points across all data scales, with its advantage becoming more pronounced as alignment degrades. Building on this insight, we develop a theoretical decision framework grounded in sample complexity and validate our findings using features extracted from ResNet-18 and CLIP ViT-B/32 on controlled datasets.

This study addresses insufficient deep integration of visual and linguistic representations. We propose FUSION-3B, a multimodal large language model featuring full-modality dynamic alignment. Methodologically, we introduce three novel components: (1) text-guided unified visual encoding, (2) context-aware recursive alignment decoding, and (3) dual-supervised semantic mapping loss—collectively transcending conventional late-fusion paradigms to enable pixel-level, question-level, and end-to-end cross-modal unified modeling. Despite its compact 3B parameter count, FUSION-3B achieves superior performance on most benchmarks using only 630 visual tokens—outperforming Cambrian-1 (8B) and Florence-VL (8B); even with only 300 visual tokens, it surpasses Cambrian-1 (8B), and it exceeds LLaVA-NeXT on over half of the evaluated benchmarks. To further support fine-grained vision–language alignment, we construct a language-driven synthetic QA dataset.

This study investigates the impact of fusion timing on the accuracy–latency trade-off in multimodal vision–language systems. We propose and systematically evaluate three fusion strategies—early, middle, and late—within a unified architecture combining BERT for language and lightweight visual backbones (MobileNetV2 or ViT) on the CMU MOSI dataset; inference latency is empirically measured on an NVIDIA Jetson Orin AGX edge platform. Results show that late fusion achieves the highest accuracy (12.3% lower MAE), while early fusion incurs the lowest latency (41.7% reduction on average), with fusion stage exhibiting a strong negative correlation between accuracy and latency. To our knowledge, this is the first work to quantitatively and systematically validate the critical influence of fusion location under a consistent experimental framework. Our findings provide reproducible architectural guidelines and empirical evidence for designing efficient multimodal models tailored to resource-constrained edge devices.

Large vision-language models (LVLMs) overly rely on deepest-layer visual features while neglecting complementary information across intermediate layers. Method: We propose an instruction-guided dynamic visual feature aggregation mechanism—the first of its kind to enable instruction-driven, adaptive selection, weighting, and cross-layer interaction of multi-depth visual features without increasing the number of visual tokens. Leveraging task-aware attention aggregation and instruction-conditioned gating, the method jointly optimizes fine-grained perception (via low-level features) and semantic understanding (via mid- to high-level features). Contribution/Results: Our approach achieves significant performance gains across 18 benchmarks spanning six diverse vision-language tasks, demonstrating both the effectiveness and strong generalizability of dynamic, hierarchical visual feature utilization.

This work challenges the prevailing view that image and text representations in vision-language models align only in deep layers. Inspired by DeepDream, the authors propose a synthesis method within an adapter-based architecture that extracts textual concept vectors layer by layer and optimizes corresponding images without auxiliary models or additional datasets. For the first time, this approach provides direct, concept-level evidence of cross-modal alignment starting from the very first layer: across seven network layers and hundreds of concepts, over 50% of images synthesized from the first layer already exhibit clear, salient visual features of target concepts—such as animals, activities, and seasons. This method offers an efficient and novel pathway toward enhancing the interpretability of vision-language models.

Existing studies lack systematic methods to dissect representational commonalities and idiosyncrasies across diverse large vision models—especially those differing in architecture and training paradigms. Method: We propose a biologically inspired, multi-dimensional representational analysis framework integrating Representational Similarity Analysis (RSA), Soft Matching, and Linear Predictivity, augmented by an improved Similarity Network Fusion (SNF) technique for cross-architectural representational similarity modeling. Contribution/Results: Our framework uncovers, for the first time, cross-architectural convergence of self-supervised models in geometric structure, unit tuning properties, and linear decodability—revealing pronounced representational alignment between hybrid architectures and masked autoencoders. The resulting robust “representation fingerprints” significantly improve model-family discrimination accuracy and expose previously unrecognized inter-model associations. Collectively, these findings establish a new paradigm for understanding how architectural inductive biases and training objectives jointly shape computational strategies in vision models.

This work addresses the limitations of existing approaches that predominantly rely on non-native late-fusion architectures and lack a systematic definition or unified framework for native multimodal modeling. The paper proposes a formal roadmap for Native Multimodal Modeling (NMM), introducing, for the first time, a rigorous formulation of “architectural nativeness.” Leveraging an input–output duality perspective, it categorizes models into three types: Multi-to-Text, Multi-to-Target, and Multi-to-Multi. A full-stack, industrial-grade NMM framework is developed, encompassing data governance, early/mid-stage fusion, end-to-end training, and deployment, all realized through a unified Transformer architecture that enables symbiotic cross-modal understanding and generation. This paradigm demonstrates superior performance and strong scalability across multiple tasks, offering a clear pathway toward truly native multimodal models.

Neural network internal representations often lack stability and cross-architectural consistency due to architectural disparities, hindering knowledge transfer and modular deployment. To address this, we propose a structured regularization framework comprising linear shaping operators and rectified path constraints, which explicitly encode inductive biases to improve geometric alignment of representations across architectures. Through theoretical analysis, controlled transfer experiments, and a novel representation alignment metric, we systematically demonstrate that structural priors significantly enhance semantic consistency among heterogeneous models. Our method improves downstream task performance in model distillation and modular learning by up to 12.3%, offering an interpretable and scalable paradigm for building robust, composable deep learning systems.

Multimodal image fusion faces two key challenges: gradient conflicts arising from cross-modal parameter sharing, which degrade performance; and modality-specific encoders that improve fusion quality yet harm task generalization. To address these, we propose a unified fusion framework integrating three novel components: semantic-aware channel pruning (to retain discriminative features), geometric affine modulation (to model inter-modal spatial discrepancies), and text-guided channel perturbation (to inject semantic priors and enhance robustness). Our method synergistically leverages pretrained semantic knowledge, channel-level perturbations, and affine transformations—achieving selective feature learning and strong cross-task generalization without introducing modality-specific parameters. Extensive experiments demonstrate consistent and significant improvements over state-of-the-art methods on major fusion benchmarks and downstream detection and segmentation tasks.

Conventional multimodal learning assumes explicit representation alignment is universally beneficial, yet its causal impact remains unverified. Method: This work systematically investigates the conditional effects of enforced alignment, proposing a controllable contrastive learning module that dynamically modulates alignment strength and establishes a quantitative relationship between alignment strength and inter-modal information redundancy—derived via information decomposition and synthetic data modeling. Contribution/Results: We demonstrate that alignment efficacy is not universal: strong alignment improves performance under high redundancy but harms modality-specific representation learning under low redundancy. To address this, we introduce a balanced mechanism that jointly preserves shared semantics and modality-specific characteristics. Empirical evaluation on both synthetic and real-world benchmarks confirms that this mechanism significantly enhances the generalization capability of unimodal encoders. Our core contribution is the formal establishment of “redundancy-dependent alignment”—a principled, interpretable, and tunable paradigm for multimodal representation learning.