Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents

2026-07-17

AI Summary

This study investigates whether intelligent agents endowed with reflection and memory capabilities can achieve observable and controllable performance gains in information extraction tasks. Focusing on structured data extraction from academic paper PDFs, we propose an enhanced agent architecture, S2, which incorporates a dynamic tool selection mechanism and an expanded set of PDF processing tools. We further introduce an evaluation framework centered on behavioral controllability. Experimental results demonstrate that the proposed agent effectively adapts its execution strategy through reflection, retrying, and memory mechanisms, significantly outperforming fixed-pipeline large language model workflows on critical failure modes, thereby validating the efficacy and superiority of our design.

Abstract

Large language model (LLM) agents are increasingly used for complex information-extraction tasks, yet it remains unclear whether agentic components such as reflection and memory lead to observable and controllable improvements over fixed LLM workflows. We study this question through conference-paper dataset extraction, where a system must identify datasets mentioned in scholarly PDFs and produce structured records. We compare a fixed workflow baseline with reflective agent variants and specify an optimized agent condition (S2) that extends the same task with richer PDF tools and dynamic tool selection. Our evaluation emphasizes process-level behavior--including tool execution, retries, reflection, memory use, runtime, and failure recovery--while treating extraction coverage and field completeness as secondary outcome measures. The paper characterizes when agentic mechanisms change system behavior, whether these changes improve task completion, and how the observed failure modes motivate an optimized agent design under the same evaluation harness.

Problem

• behavioral controllability
• agentic models
• information extraction
• LLM agents
• fixed workflows

Innovation

• reflective agents
• behavioral controllability
• dynamic tool selection
• information extraction
• LLM agents