On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrieval

2025-06-13

AI Summary

This study addresses multimodal dialogue response retrieval—jointly retrieving text and image responses for dialogue contexts. To overcome limitations in existing approaches regarding cross-modal alignment and subtask coordination, we propose two integration paradigms: a two-stage pipeline and an end-to-end joint modeling framework, and conduct the first systematic comparison of their performance. We innovatively introduce parameter-sharing mechanisms across modalities and across subtasks, integrating contrastive learning and joint optimization within both dual-encoder and end-to-end architectures. Experimental results demonstrate that the end-to-end approach achieves comparable retrieval accuracy to the two-stage method while featuring greater architectural simplicity. Moreover, our parameter-sharing strategy improves retrieval accuracy by 3.2% and reduces model parameters by 37%. These findings establish a new, efficient, and lightweight paradigm for multimodal dialogue response retrieval.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal chatbots have become one of the major topics for dialogue systems in both research community and industry. Recently, researchers have shed light on the multimodality of responses as well as dialogue contexts. This work explores how a dialogue system can output responses in various modalities such as text and image. To this end, we first formulate a multimodal dialogue response retrieval task for retrieval-based systems as the combination of three subtasks. We then propose three integration methods based on a two-step approach and an end-to-end approach, and compare the merits and demerits of each method. Experimental results on two datasets demonstrate that the end-to-end approach achieves comparable performance without an intermediate step in the two-step approach. In addition, a parameter sharing strategy not only reduces the number of parameters but also boosts performance by transferring knowledge across the subtasks and the modalities.

Problem

• Effectiveness of integration methods for multimodal response retrieval
• Comparing two-step and end-to-end multimodal dialogue approaches
• Parameter sharing strategy for performance and efficiency improvement

Innovation

• Two-step and end-to-end multimodal integration methods
• Parameter sharing strategy reduces parameters
• End-to-end approach achieves comparable performance