EndoAgent: A Memory-Guided Reflective Agent for Intelligent Endoscopic Vision-to-Decision Reasoning

2025-08-10

AI Summary

Existing endoscopic AI systems lack cross-task coordination capabilities, hindering their support for complex clinical decision-making workflows. To address this, we propose EndoAgent—the first intelligent reasoning agent framework specifically designed for colonoscopic diagnosis, integrating multimodal perception, expert tool orchestration, and a unified reasoning loop. We introduce a novel memory-guided dual-memory mechanism—comprising short-term action tracking and long-term experience modeling—to enhance logical coherence and reasoning robustness. Additionally, we construct EndoAgentBench, a dedicated benchmark for evaluating endoscopic reasoning agents. Evaluated on 5,709 real-world clinical cases, EndoAgent significantly outperforms both general-purpose and medical multimodal baseline models. It demonstrates superior flexibility, interpretability, and clinical adaptability in multi-step diagnostic tasks, establishing a new foundation for clinically grounded, agent-based endoscopic intelligence.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent/AI Theories and Architectures
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Developing general artificial intelligence (AI) systems to support endoscopic image diagnosis is an emerging research priority. Existing methods based on large-scale pretraining often lack unified coordination across tasks and struggle to handle the multi-step processes required in complex clinical workflows. While AI agents have shown promise in flexible instruction parsing and tool integration across domains, their potential in endoscopy remains underexplored. To address this gap, we propose EndoAgent, the first memory-guided agent for vision-to-decision endoscopic analysis that integrates iterative reasoning with adaptive tool selection and collaboration. Built on a dual-memory design, it enables sophisticated decision-making by ensuring logical coherence through short-term action tracking and progressively enhancing reasoning acuity through long-term experiential learning. To support diverse clinical tasks, EndoAgent integrates a suite of expert-designed tools within a unified reasoning loop. We further introduce EndoAgentBench, a benchmark of 5,709 visual question-answer pairs that assess visual understanding and language generation capabilities in realistic scenarios. Extensive experiments show that EndoAgent consistently outperforms both general and medical multimodal models, exhibiting its strong flexibility and reasoning capabilities.

Problem

• Developing AI for endoscopic image diagnosis coordination
• Enhancing multi-step clinical workflow handling in endoscopy
• Integrating reasoning with adaptive tool selection in endoscopy

Innovation

• Memory-guided agent for endoscopic analysis
• Dual-memory design for decision-making
• Integrated expert tools in reasoning loop