DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search

2026-07-29

AI Summary

This work addresses the poor reproducibility and limited multilingual transferability of existing retrieval models, which often rely on closed-source data. We propose a fully open-source, end-to-end training framework leveraging a reconstructed dataset of 665 million English contrastive pairs and 1.88 million supervised pairs to train both DenseOn (a single-vector dense retriever) and LateOn (a ColBERT-style late-interaction model). These models are extended to eight languages, marking the first public release of large-scale multilingual retrieval data and models. Experimental results show that LateOn significantly outperforms DenseOn on unseen languages, achieving average nDCG@10 scores of 56.20 and 57.22 respectively on BEIR—setting new state-of-the-art results at this scale. Furthermore, our analysis reveals that translate-train functions as a general multilingual generalization mechanism rather than merely a target-language expansion strategy. All code, data, and models are publicly released.

Abstract

State-of-the-art retrieval models increasingly rely on closed training data, creating a reproducibility gap. We present an open end-to-end recipe for training retrieval models and study how English supervision transfers to multilingual retrieval through translate-train. We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources and build 1.88M supervised fine-tuning pairs with mined hard negatives. Training yields two 149M-parameter models: DenseOn, a single-vector dense model, and LateOn, a ColBERT-style late-interaction model. They achieve 56.20 and 57.22 average nDCG@10 on BEIR, respectively, setting new state-of-the-art results for this size class. We then translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples, and train mDenseOn and mLateOn, two 307M-parameter models built on mmBERT-base. Despite sharing their backbone, data, and objectives, their representations behave differently: the dense model is strong on English and translated languages but degrades outside translate-train support, whereas the late-interaction model generalizes better to unseen languages and scripts. This suggests that token-level matching turns translate-train from a target-language expansion strategy into a multilingual generalization recipe. We publicly release the models, datasets, and training code.

Problem

• retrieval models
• multilingual search
• translate-train
• reproducibility gap
• cross-lingual generalization

Innovation

• late-interaction
• translate-train
• open retrieval models
• multilingual generalization
• dense retrieval