Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation

2026-07-19

AI Summary

This work addresses the challenges of target ambiguity and inter-frame inconsistency in user-centric interactive medical image segmentation, which arise from semantic ambiguity and visual variability. To tackle these issues, the authors propose EgoMed-Agent, a multi-agent system that introduces a novel target confirmation mechanism based on reliability scoring and a localization-guided mask propagation strategy. By integrating dual workflows—target confirmation and localization-guided propagation—the system enables precise and temporally consistent segmentation through effective human–AI collaboration. The method leverages a multi-agent architecture, instruction–target alignment scoring, and a tightly coupled mechanism that jointly optimizes per-frame localization and mask propagation. Evaluated on first-person interactive medical image segmentation, EgoMed-Agent achieves an average Dice score of 71.34%, substantially outperforming the best text-prompt baseline by 11.70%.

Abstract

Interactive egocentric medical image segmentation (IEMIS) plays an important role in smart-glasses-assisted medical image review, segmenting the medical targets a clinician refers to from their egocentric view. Once it succeeds, the object-level visual evidence it provides strengthens the review and underpins fine-grained analysis and clinical decision-making. However, the instruction and the video both come from the user's egocentric perspective, which poses two challenges. (1) Semantic ambiguity leaves the model unable to confirm the user-intended target. (2) Visual variability makes the segmentation jump from frame to frame. In this paper, we propose EgoMed-Agent, a multi-agent system that understands the target from the human perspective through two workflows. (1) The \textit{Target Confirmation Workflow} grounds the instruction against candidate targets with a reliability score, confirming the target when the grounding is reliable and asking the user to clarify when it is not, thereby confirming the segmentation target. (2) The \textit{Localization-Guided Propagation Workflow} couples mask propagation with per-frame target localization, using the localized target to correct the propagated mask whenever the two diverge, so the segmentation stays on the target across the egocentric video. Extensive experiments show that EgoMed-Agent reaches 71.34\% average Dice, far above the best text-prompted baseline (11.70\%). Our code is available at \href{https://github.com/wdyyyyyy/EgoMed-Agent}{our project page}.

Problem

• interactive egocentric medical image segmentation
• semantic ambiguity
• visual variability
• first-person vision
• medical image analysis

Innovation

• multi-agent system
• egocentric medical image segmentation
• target confirmation
• localization-guided propagation
• interactive segmentation