From Weak Task Specifications to Scientific Extraction Agents: Optimizing Task Construction

2026-09-28

AI Summary

This study addresses the high cost of manually constructing task configurations for scientific information extraction and the difficulty of automatically generating complete pipelines from brief objectives alone. To overcome these challenges, this work proposes an end-to-end framework that formulates task construction as a series of optimizable components. Methodologically, schemas, instructions, and evaluation criteria are automatically generated from weak specifications, while textual gradient feedback and failure-focused update mechanisms are introduced to enable joint optimization, ensuring the entire pipeline remains fully editable. Experiments conducted on a heterogeneous corpus of catalysis literature demonstrate that the joint optimization of schemas and instructions consistently yields superior performance across all settings, a finding further corroborated by blind human evaluations.

Abstract

Most methods that optimize LLM prompts and agent workflows assume that task-specific output schemas, extraction instructions, and evaluation criteria are predefined. For scientific extraction agents, however, a short task goal may not fully determine these components, while specifying them manually is costly. We study the upstream problem of constructing the task-specific configuration from a weak specification containing only a short goal and unannotated reference documents. Rather than treating automatic construction as a fixed preprocessing step, our framework constructs a task-specific schema, extraction instructions, and base training rubrics, then keeps schema construction and extraction instructions editable during optimization. Failure-focused updates concentrate textual-gradient feedback on lower-scoring documents, while training-time evaluation criteria adapt to recurring failures. On a heterogeneous-catalysis literature corpus, automatic construction remains improvable, and optimizing both schema construction and extraction instructions performs best across all four judge-rubric settings, with ablations and blinded human evaluation supporting the proposed formulation.

Problem

• scientific extraction agents
• weak task specifications
• task construction
• output schema
• LLM agents

Innovation

• Scientific Extraction Agents
• Weak Task Specifications
• Textual-Gradient Feedback
• Schema Construction Optimization
• Failure-Focused Updates