Multimodal Search in Chemical Documents and Reactions

2025-02-24

AI Summary

This work addresses the challenge of cross-modal retrieval among chemical reactions, molecular structures, and textual descriptions in scientific literature. We propose the first end-to-end multimodal retrieval system supporting joint queries over molecular graphs, reaction schemes, and natural language. Our method integrates chemistry-aware graph OCR for structural recognition, structured extraction of reaction information from both tabular and free-text sources, and a chemically grounded multimodal embedding framework with semantic alignment—enabling precise cross-modal matching among graph, text, and table modalities within a unified indexing architecture. The key contribution is the first full-stack, domain-specific multimodal alignment and retrieval framework for chemistry, which significantly improves accuracy and interpretability in complex reaction retrieval. Expert evaluation confirms its effectiveness in real-world research scenarios involving hybrid modality exploration.

Technology Category

Machine Learning: Multimodal LearningComputer Vision: Multi-modal VisionIntelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGGraph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphsSemantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

We present a multimodal search tool that facilitates retrieval of chemical reactions, molecular structures, and associated text from scientific literature. Queries may combine molecular diagrams, textual descriptions, and reaction data, allowing users to connect different representations of chemical information. To support this, the indexing process includes chemical diagram extraction and parsing, extraction of reaction data from text in tabular form, and cross-modal linking of diagrams and their mentions in text. We describe the system's architecture, key functionalities, and retrieval process, along with expert assessments of the system. This demo highlights the workflow and technical components of the search system.

Problem

• Facilitates retrieval of chemical reactions
• Combines molecular diagrams and text
• Supports cross-modal linking of chemical information

Innovation

• Multimodal search tool
• Chemical diagram extraction
• Cross-modal linking system