Pailitao-MMSearch: Building Native E-Commerce Multimodal Search Foundation

2026-07-19

AI Summary

This work addresses the challenge that existing e-commerce search systems struggle to effectively interpret complex cross-modal queries combining images, text, and speech, due to fragmented single-modality models and the lack of fine-grained semantic understanding in general-purpose vision-language models. To bridge this gap, we propose a native multimodal foundation model tailored for e-commerce, built upon the Qwen large language model with a unified architecture. Our approach introduces a novel Hybrid Semantic ID (HybSID), a two-stage continual pre-training strategy, and a hybrid inference-aware post-training mechanism to achieve precise multimodal semantic alignment and accurate modeling of commercial intent. Deployed on Taobao’s Pailitao platform, the model demonstrates significant gains in online A/B tests, yielding a 13.61% increase in GMV and an 8.21% rise in transaction volume over existing solutions.

Abstract

The evolution of e-commerce has fundamentally transformed how users search for products, shifting from simple text-based keyword queries to complex multimodal interactions that seamlessly combine product images, natural language descriptions, and mixed-intent instructions. However, existing approaches face a critical dilemma: single-modal specialist models, deployed independently for text retrieval, visual search, and voice recognition, operate in isolation and cannot handle cross-modal queries, while general-purpose vision-language models lack the domain-specific knowledge necessary for fine-grained product understanding, user behavior modeling, and commercial intent reasoning. In this work, we present Pailitao-MMSearch, one native e-commerce multimodal search foundation model designed to bridge this gap. Our approach introduces three key innovations: (1)HybSID (Hybrid Semantic ID);(2)a two-stage continual pre-training strategy; and (3)a hybrid reasoning post-training pipeline. Built upon Qwen and deployed on Taobao's Pailitao multimodal search platform, Pailitao-MMSearch achieves substantial improvements in online A/B testing, including up to +13.61\% in Gross Merchandise Volume (GMV) and +8.21\% in transaction volume compared to traditional multi-modal search pipeline, demonstrating the effectiveness of our native e-commerce multimodal search large language models.

Problem

• multimodal search
• e-commerce
• cross-modal query
• domain-specific knowledge
• product understanding

Innovation

• HybSID
• continual pre-training
• multimodal search
• e-commerce foundation model
• hybrid reasoning