EgoAfford: Task-Oriented Affordance Grounding via Egocentric Referring Segmentation

2026-08-05

AI Summary

This work addresses the challenge of identifying action-relevant functional regions—such as direct objects, tools, and target locations—from first-person visual observations in complex multi-step tabletop tasks and generating subsequent plans. To this end, the authors propose a novel paradigm that unifies task state representation, semantic role labeling, and functional region grounding through action-role-conditioned referring segmentation. They introduce EgoAfford, a new multimodal benchmark comprising semantically aligned image sequences of multi-step tasks paired with high-level instructions, along with EgoLens, a dedicated 3B-parameter multimodal large language model equipped with role-specific mask decoders. Experiments demonstrate that EgoLens achieves state-of-the-art performance on both synthetic and real-world data, validating the efficacy of jointly modeling perception and planning, and establishing a new benchmark and reference model for embodied intelligence research.

Abstract

Part-level affordance grounding has advanced the localization of functional object regions associated with elemental actions. Extending this capability to complex tasks calls for connecting the semantic roles of participating objects with task-state-aligned visual observations and multi-step planning. We introduce EgoAfford, a benchmark designed to connect these three aspects. Given an egocentric observation and a high-level tabletop task, a model must generate the remaining plan and segment the functional regions of up to three components of the next action: the direct object, instrument, and destination. EgoAfford comprises approximately 15.5k human-verified images from 2,000 generated multi-step scenes, organized as semantically aligned, task-complete image series, together with EgoAfford-Real, 102 manually captured images spanning 26 tasks. We further present EgoLens, a 3B multimodal large language model with role-specific mask decoders, as an in-domain reference model for this joint task. Evaluations of recent referring-segmentation MLLMs, commercial-VLM--SAM2 pipelines, and EgoLens highlight the complementary challenges of next-step inference and action-role-conditioned part grounding. EgoLens establishes strong reference performance on both generated and manually captured observations. Together, EgoAfford and EgoLens provide a foundation for jointly studying perception and planning in multi-step tabletop tasks. Our project page is available at: https://egoafford.github.io

Problem

• affordance grounding
• egocentric vision
• task-oriented segmentation
• multi-step planning
• referring segmentation

Innovation

• affordance grounding
• egocentric vision
• referring segmentation
• multimodal LLM
• task-oriented perception