Benchmarking Agentic Systems in Automated Scientific Information Extraction with ChemX

2025-10-01

AI Summary

Chemical information extraction faces significant challenges due to textual heterogeneity—including domain-specific terminology, tabular structures, and contextual ambiguity—limiting the performance of existing agent-based approaches. To address this, we introduce ChemX, the first authoritative benchmark specifically designed for chemical scientific information extraction. ChemX comprises 10 expert-validated datasets covering nanomaterials and small molecules, enabling systematic and reproducible evaluation. Methodologically, we propose a single-agent architecture with controlled document preprocessing, integrating large language model agents (ChatGPT Agent, GPT-5, and reasoning-enhanced GPT-5 Thinking). Evaluation is conducted via fine-grained analysis grounded in expert annotations. Experiments uncover critical deficiencies in current systems—particularly in chemical term comprehension, table parsing, and semantic disambiguation. ChemX fills a longstanding gap by establishing the first standardized, scalable, and high-fidelity evaluation infrastructure for chemical information extraction, facilitating rigorous algorithmic development and cross-model comparison.

Technology Category

• Natural Language Processing: Information Extraction
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The emergence of agent-based systems represents a significant advancement in artificial intelligence, with growing applications in automated data extraction. However, chemical information extraction remains a formidable challenge due to the inherent heterogeneity of chemical data. Current agent-based approaches, both general-purpose and domain-specific, exhibit limited performance in this domain. To address this gap, we present ChemX, a comprehensive collection of 10 manually curated and domain-expert-validated datasets focusing on nanomaterials and small molecules. These datasets are designed to rigorously evaluate and enhance automated extraction methodologies in chemistry. To demonstrate their utility, we conduct an extensive benchmarking study comparing existing state-of-the-art agentic systems such as ChatGPT Agent and chemical-specific data extraction agents. Additionally, we introduce our own single-agent approach that enables precise control over document preprocessing prior to extraction. We further evaluate the performance of modern baselines, such as GPT-5 and GPT-5 Thinking, to compare their capabilities with agentic approaches. Our empirical findings reveal persistent challenges in chemical information extraction, particularly in processing domain-specific terminology, complex tabular and schematic representations, and context-dependent ambiguities. The ChemX benchmark serves as a critical resource for advancing automated information extraction in chemistry, challenging the generalization capabilities of existing methods, and providing valuable insights into effective evaluation strategies.

Problem

• Benchmarking agent-based systems for chemical data extraction challenges
• Evaluating automated extraction methods for nanomaterials and molecules
• Addressing domain-specific terminology and complex representation issues

Innovation

• Introduced ChemX benchmark datasets for chemical extraction
• Developed single-agent approach for document preprocessing control
• Evaluated agentic systems against modern baselines like GPT-5