Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations

2025-05-27

AI Summary

Current large language models (LLMs) lack systematic evaluation of structured, constraint-aware reasoning capabilities in chemistry—particularly for molecular property optimization and reaction prediction. Method: We propose ChemCoTBench, the first benchmark framework for “slow-thinking” chemical reasoning, introducing a novel “modular chemical operation” paradigm (addition, deletion, substitution) that formalizes molecular transformations as interpretable, stepwise symbolic processes. It integrates graph-structured molecular representation, constraint-driven chain-of-thought (CoT) reasoning, and a manually curated dataset. Contribution/Results: ChemCoTBench establishes an evaluation framework measuring both reasoning-path traceability and rule adherence. Experiments demonstrate substantial improvements in LLMs’ ability to model domain-specific constraints and reaction rules, advancing trustworthy, interpretable chemical AI.

Technology Category

Knowledge Representation and Reasoning: Computational Complexity of ReasoningCognitive Modeling & Cognitive Systems: Conceptual Inference and ReasoningConstraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

While large language models (LLMs) with Chain-of-Thought (CoT) reasoning excel in mathematics and coding, their potential for systematic reasoning in chemistry, a domain demanding rigorous structural analysis for real-world tasks like drug design and reaction engineering, remains untapped. Current benchmarks focus on simple knowledge retrieval, neglecting step-by-step reasoning required for complex tasks such as molecular optimization and reaction prediction. To address this, we introduce ChemCoTBench, a reasoning framework that bridges molecular structure understanding with arithmetic-inspired operations, including addition, deletion, and substitution, to formalize chemical problem-solving into transparent, step-by-step workflows. By treating molecular transformations as modular"chemical operations", the framework enables slow-thinking reasoning, mirroring the logic of mathematical proofs while grounding solutions in real-world chemical constraints. We evaluate models on two high-impact tasks: Molecular Property Optimization and Chemical Reaction Prediction. These tasks mirror real-world challenges while providing structured evaluability. By providing annotated datasets, a reasoning taxonomy, and baseline evaluations, ChemCoTBench bridges the gap between abstract reasoning methods and practical chemical discovery, establishing a foundation for advancing LLMs as tools for AI-driven scientific innovation.

Problem

• Evaluating LLMs' chemical reasoning beyond simple QA tasks
• Addressing lack of step-by-step reasoning in molecular optimization
• Bridging abstract reasoning with practical chemical discovery challenges

Innovation

• Modular chemical operations for molecular transformations
• Step-by-step reasoning with ChemCoTBench framework
• Bridges molecular structure understanding with arithmetic operations