ProtStructQA: A Denotation Threshold in Protein Structural Reasoning

2026-05-29

AI Summary

Existing protein language models struggle to accurately map natural language questions to executable 3D structural measurements. This work introduces ProtStructQA—the first protein structure question-answering benchmark grounded in an executable domain-specific language (DSL)—where questions and answers are generated by running latent DSL programs on AlphaFold-predicted structures. The study reframes scientific question answering as a “language-to-structure-measurement” compilation process, integrating grammar-constrained voting, chain-of-thought reasoning, and ReAct-style tool invocation. Evaluation on 382.2K questions reveals a referential competence threshold in Qwen3 and Gemma-3 models within the 1.7B–4B parameter range, which significantly influences the effectiveness of their reasoning strategies.

Abstract

Protein-language systems are often evaluated by whether they generate plausible biological text, but a structural question has a sharper semantics: it denotes a measurement in a 3D coordinate system. We introduce ProtStructQA, an executable benchmark for protein structural question answering in which each natural-language question is generated from a hidden typed domain-specific language (DSL) program and the answer is obtained by executing that program on an AlphaFold-predicted structure. ProtStructQA releases 382.2K questions covering confidence, distances, predicted aligned error (PAE), solvent exposure, secondary structure, topology and contacts, and held-out compositions: a 330K active benchmark over 10K proteins from four species, plus a 52.2K hard-negative robustness pool. Without fine-tuning, we evaluate Qwen3 models from 0.6B to 8B under direct prompting, chain-of-thought, grammar-constrained executable voting, executable voting with chain-of-thought, and multi-turn ReAct-style tool use, and replicate the headline finding on Gemma-3-1B and Gemma-3-12B. We find a capability-dependent denotation threshold between Qwen3-1.7B and Qwen3-4B: below it, tool-mediated ReAct dominates because models often fail to produce executable denotations; above it, chain-of-thought flips from mostly harmful to strongly beneficial and becomes the strongest strategy on most splits. Parse-failure and family-level analyses show that the threshold is a transition from unparseable language to executable structural denotation, while grammar and execution remain selectively valuable for PAE and secondary-structure queries. ProtStructQA reframes scientific QA as compilation from language to measurement and provides a diagnostic testbed for when language models can map words to executable 3D structural measurements.

Problem

• protein structural reasoning
• executable benchmark
• structural question answering
• denotation threshold
• 3D coordinate measurement

Innovation

• executable reasoning
• protein structural QA
• denotation threshold
• domain-specific language
• 3D structural measurement