embedding concatenation

Deploying embedding models in resource-constrained environments (e.g., browsers, edge devices) faces a fundamental trade-off between deployment feasibility and performance—lightweight models suffer from weak representational capacity, while compact deployment of large models remains challenging. Method: We propose Concat-Encode-Quantize—a novel framework that concatenates raw embeddings from multiple lightweight base models, applies a lightweight shared decoder for unified dimensionality reduction, and trains end-to-end using Matryoshka Representation Learning (MRL) loss—without fine-tuning any base model. Contribution/Results: The framework significantly enhances representation robustness under aggressive compression and quantization. On the MTEB retrieval subset, a four-model ensemble achieves 89% of the original performance after 48× quantization, substantially outperforming a single large model of comparable total parameter count. This yields an efficient, robust, and production-ready embedding solution for dense retrieval and semantic search.

This work addresses the lack of principled understanding in existing literature regarding the choice between cross-attention and feature concatenation strategies for multimodal fusion, which has largely relied on empirical heuristics. Through controlled experiments and theoretical analysis, we demonstrate for the first time that feature alignment quality is the key determinant of fusion strategy performance: under pre-aligned features, concatenation consistently outperforms cross-attention by 4.1–5.1 percentage points across all data scales, with its advantage becoming more pronounced as alignment degrades. Building on this insight, we develop a theoretical decision framework grounded in sample complexity and validate our findings using features extracted from ResNet-18 and CLIP ViT-B/32 on controlled datasets.

To address the tension between limited device memory and performance preservation in large model deployment, this paper proposes a functional redundancy compression method targeting Transformer feed-forward sublayers, departing from conventional pruning paradigms. Our core innovation lies in the first systematic alignment and fusion of feed-forward sublayers exhibiting high activation similarity, achieved via a joint weight-activation similarity metric for sublayer alignment and a weighted fusion strategy. We validate generalizability across diverse tasks—including language modeling, image classification, and machine translation. On ViT, our method removes 21% of parameters while retaining 99% of original accuracy; fusing over one-third of feed-forward sublayers yields accuracy on par with the full model—significantly outperforming strong pruning baselines. This work establishes a novel paradigm for efficient large-model lightweighting through functional redundancy exploitation rather than structural sparsification.

To address the low representation fusion efficiency of speech foundation models in downstream tasks, this paper proposes the first unified framework integrating both model-level and layer-level fusion. The core innovation is a lightweight cross-model interface module that enables joint alignment and end-to-end optimization across multiple upstream speech models—whether self-supervised or supervised—as well as their internal layer-wise features. This module facilitates deep semantic integration across heterogeneous models, significantly enhancing fusion flexibility and representational capacity. Evaluated on automatic speech recognition (ASR) and language-like analysis tasks, the method consistently outperforms existing fusion strategies. It demonstrates strong scalability and consistent performance gains across varying model scales and numbers of constituent models, establishing a new state-of-the-art in speech representation fusion.

The evolution of embedding techniques from word vectors to multimodal representations remains fragmented, lacking a unified framework that integrates advances across linguistic, cross-lingual, personalized, and multimodal domains—particularly for embodied multimodal learning in large language models. Method: We systematically survey static and contextual language representations, cross-lingual and personalized modeling, sentence/document embeddings, and multimodal fusion in vision, robotics, and cognitive science. We synthesize recent progress in interpretability, model compression, numerical encoding, and bias mitigation, and propose a novel paradigm emphasizing strong alignment across non-textual modalities and scalable training. Contributions: We construct a comprehensive knowledge graph of end-to-end embedding technologies—from Word2Vec and BERT to GPT, generative topic models, and multimodal alignment/distillation methods—identifying key technical bottlenecks and ethical challenges. This work delivers the first systematic roadmap for multimodal, embodied learning in foundation models.

This work addresses the challenge of efficiently combining the capabilities of multiple fine-tuned large language models without resorting to costly retraining or computationally expensive ensembles. The authors propose FUSE, a four-dimensional taxonomy that systematically unifies the theoretical foundations of model merging—such as loss landscape geometry and mode connectivity—with practical strategies including weight averaging, task vector arithmetic, sparsity-enhanced methods, mixture-of-experts architectures, and evolutionary optimization. By establishing a comprehensive, training-free framework for model merging, this study organizes existing open-source tools and evaluation benchmarks, while also identifying critical theoretical gaps, scalability challenges, and the need for standardization. The resulting knowledge体系 offers a structured roadmap to guide future research in efficient and effective model fusion.

Existing multimodal embedding models struggle to uniformly support text, images, video, and audio, often omitting audio or covering only a subset of modalities. This work proposes a unified four-modality embedding space built upon a frozen vision–language foundation model, augmented with a lightweight audio tower connector and modality-gated deep adapters, requiring no updates to the base model parameters. By aligning only audio with text, the approach enables cross-modal retrieval across all pairs—including audio–image—while fully preserving the original performance on pre-existing modalities. Both generations of the proposed model are trained within hours on a single GPU and achieve strong results in audio–text and audio–image retrieval. The code, model weights, and evaluation tools are publicly released.

This study investigates the impact of fusion mechanisms between positional encoding and token embeddings on Transformer performance in long-sequence tasks. By systematically comparing additive, concatenation-with-projection, and scalar-gated strategies within a unified architecture—and further introducing a lightweight convolutional gating module to inject local inductive bias—the work demonstrates that learnable fusion approaches significantly outperform conventional additive fusion. Comprehensive evaluations, including controlled ablation studies, paired random-seed analyses, cross-dataset validation, and experiments across multiple positional encoding families, consistently reveal that the choice of fusion mechanism is a critical design factor for long-range modeling. The proposed method yields stable performance gains and exhibits strong generalization, particularly on long-text classification benchmarks such as ArXiv.

This study investigates whether robust multilingual performance can be achieved by merging monolingual pretrained models. Addressing the limitations of existing approaches in cross-lingual transfer, we systematically compare mixed multilingual pretraining, independent monolingual pretraining, and various model merging strategies. Our experiments reveal that while monolingual models excel in their respective languages, naive merging leads to severe performance collapse, underscoring the critical dependence of successful merging on cross-lingual representational similarity. We further demonstrate, for the first time, that merging mechanisms effective during fine-tuning do not readily generalize to the language-specific pretraining phase. These findings provide crucial theoretical insights and practical warnings for the design and construction of multilingual models.

This work addresses the limitations of conventional model fusion approaches—such as parameter averaging—which often introduce non-generalizable features when source models differ substantially and lack theoretical grounding or effective metrics for assessing fusion compatibility. The study establishes, for the first time, a theoretical connection between model fusion and ensemble learning, and introduces M-Loss, a layer- or neuron-level inconsistency metric computed using only a small amount of unlabeled data to quantify the discrepancy between parameter averaging and model ensembling. This metric effectively guides the optimization of fusion strategies and the evaluation of parameter importance, thereby enhancing pruning efficiency. Experiments demonstrate that M-Loss significantly improves alignment between fused and ensemble models, enabling efficient and accurate model integration while reducing inference cost and storage overhead.