Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models

2025-10-15

AI Summary

To address insufficient visual similarity modeling in e-commerce product recommendation, this work pioneers the integration of the SigLIP vision-language model into recommender systems, proposing a cross-modal semantic alignment framework grounded in image–text contrastive learning. We fine-tune SigLIP using a sigmoid-based contrastive loss and design a lightweight image encoder to generate highly discriminative item embeddings, thereby enhancing visual representation capability. Offline evaluation demonstrates a 9.1% improvement in nDCG@5; online A/B testing shows a 50% increase in click-through rate and a 14% uplift in conversion rate. This study not only validates SigLIP’s effectiveness for e-commerce recommendation but also establishes a practical, business-oriented vision–language joint modeling paradigm. It provides a reusable technical pathway for multimodal recommendation, bridging the gap between foundational vision-language models and real-world industrial applications.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

On large-scale e-commerce platforms with tens of millions of active monthly users, recommending visually similar products is essential for enabling users to efficiently discover items that align with their preferences. This study presents the application of a vision-language model (VLM) -- which has demonstrated strong performance in image recognition and image-text retrieval tasks -- to product recommendations on Mercari, a major consumer-to-consumer marketplace used by more than 20 million monthly users in Japan. Specifically, we fine-tuned SigLIP, a VLM employing a sigmoid-based contrastive loss, using one million product image-title pairs from Mercari collected over a three-month period, and developed an image encoder for generating item embeddings used in the recommendation system. Our evaluation comprised an offline analysis of historical interaction logs and an online A/B test in a production environment. In offline analysis, the model achieved a 9.1% improvement in nDCG@5 compared with the baseline. In the online A/B test, the click-through rate improved by 50% whereas the conversion rate improved by 14% compared with the existing model. These results demonstrate the effectiveness of VLM-based encoders for e-commerce product recommendations and provide practical insights into the development of visual similarity-based recommendation systems.

Problem

• Improving visual product recommendations using vision-language models
• Fine-tuning SigLIP model for e-commerce item embeddings
• Enhancing click-through and conversion rates through visual similarity

Innovation

• Fine-tuned SigLIP vision-language model for recommendations
• Developed image encoder generating item embeddings
• Applied contrastive learning with product image-title pairs