affordance detection

Current visual affordance prediction research suffers from fragmented task definitions—such as grasp detection and affordance classification—each redefining “affordance” independently, leading to incomparable benchmarks and poor reproducibility, thereby hindering generalization in robotic interaction. To address this, we propose a unified modeling framework: (1) formally define visual affordance as a mapping from object visual features to physical attributes (e.g., mass) and subsequently to task-relevant interactions; (2) introduce the *Affordance Sheet*, a standardized documentation protocol specifying data curation, evaluation metrics, and implementation details; and (3) establish a cross-task benchmark and reproducibility diagnostic toolkit integrating visual perception, physical attribute estimation, and interaction modeling. This work is the first to achieve conceptual unification, evaluation consistency, and implementation transparency in affordance prediction, significantly improving model reliability and generalization in real-world robotic settings.

Recent advances in large language models have led to strong performance on reasoning and environment-interaction tasks, yet their ability for creative problem-solving remains underexplored. We study this capability through the lens of creative tool use, where a model repurposes available objects by reasoning about their affordances and attributes rather than relying on canonical usage. As a first step, we introduce CreativityBench, a benchmark for evaluating affordance-based creativity in LLMs. To this end, we build a large-scale affordance knowledge base (KB) with 4K entities and 150K+ affordance annotations, explicitly linking objects, parts, attributes, and actionable uses. Building on this KB, we generate 14K grounded tasks that require identifying non-obvious yet physically plausible solutions under constraints. Evaluations across 10 state-of-the-art LLMs, including closed and open-source models, show that models can often select a plausible object, but fail to identify the correct parts, their affordances, and the underlying physical mechanism needed to solve the task, leading to a significant drop in performance. Furthermore, improvements from model scaling quickly saturate, strong general reasoning does not reliably translate to creative affordance discovery, and common inference-time strategies such as Chain-of-Thought yield limited gains. These results suggest that creative tool use remains a major challenge for current models, and that CreativityBench provides a useful testbed for studying this missing dimension of intelligence, with potential implications for planning and reasoning modules in future agents.

To address the limited generalization of affordance grounding models to unseen objects and novel affordances in embodied intelligence, this paper introduces the first SAM-based extension framework for functional region identification. Methodologically, it designs an affordance-adaptive module and establishes a coarse-to-fine multi-stage supervised training paradigm, enabling end-to-end generation of affordance heatmaps and facilitating the transfer of foundational segmentation models into functional semantic space. Crucially, it aligns vision foundation models with functional semantics, supporting zero-shot affordance localization. Evaluated on the AGD20K benchmark, the approach significantly outperforms state-of-the-art methods and demonstrates strong generalization to both unseen objects and novel affordances. This work provides a scalable, function-aware perception foundation for open-world interaction in embodied AI systems.

This work investigates the affordance generalization capabilities of vision–language–action (VLA) models when encountering novel objects that possess familiar physical properties but have never been seen before. To this end, we introduce BusyBox—a physical evaluation benchmark built upon six interchangeable modules, which enables the systematic and semi-automated assessment of model performance by generating visually diverse yet affordance-consistent object variants through module rotation and substitution. BusyBox provides, for the first time, a reproducible, low-cost, and easily constructible real-world testbed, accompanied by open-sourced 3D printing schematics, an electronic bill of materials, and a dual-arm robot demonstration dataset. Experiments reveal that current state-of-the-art open-source VLA models, such as π₀.₅ and GR00T-N1.6, exhibit limited generalization on this benchmark, thereby validating its effectiveness and necessity.

Existing approaches struggle to accurately localize multifunctional regions in cluttered real-world scenes according to task instructions and lack benchmarks supporting complex mappings such as one-to-many task-to-region correspondences. This work proposes a task-conditioned, scene-level functional affordance localization framework and introduces the first real-scene affordance benchmark dataset encompassing both single- and multi-region instruction mappings. To enable efficient and high-quality annotation, we design the A2A-AffordGen pipeline, which integrates large language model filtering, interactive segmentation, mask refinement, and human verification. Models trained on this dataset significantly outperform baseline methods—including general-purpose segmentation models, vision-language models, and affordance distillation approaches—in both task-level localization accuracy and spatial priors for downstream manipulation tasks.

This work addresses a critical limitation in existing affordance reasoning benchmarks, which inadvertently reveal object identities and thereby encourage models to rely on memorization rather than physical properties for inference. To overcome this, the authors introduce Affordance20Q, a novel benchmark that evaluates a model’s ability to infer 59 distinct affordances based solely on object attributes such as shape and material, while concealing object identity through a 20-question yes/no guessing game. The benchmark comprises 454 objects and 1,009 games. Furthermore, the study proposes Knowledge-base Anchored Rule Induction (KARI), a new method that leverages external knowledge to enhance rule-based reasoning. Experimental results demonstrate that current large language models underperform humans by approximately 20 points on this task, yet KARI improves the performance of open-source models by up to 15.2 points.

This work addresses the limited generalization of existing robotic planning systems, which rely heavily on visual appearance and neglect task-relevant functional properties, thereby struggling in novel robot-object interaction scenarios. To overcome this, the authors propose A4D, a novel approach that establishes a shared latent space centered on functional attributes (e.g., “movable”). A4D enables efficient reasoning through alignment of visual and functional embeddings and proximity-driven matching to functional prototypes. Furthermore, it incorporates an uncertainty-based few-shot mechanism for discovering new functions in previously unseen contexts. The method achieves 94% accuracy in known-function inference—surpassing prior approaches by over 15 percentage points—and, for novel functions, requires less than 10% of the training data to boost accuracy from 70% to above 90%, while also delivering a hundredfold speedup in inference time.

This work proposes the first open-world general-purpose affordance foundation model, unifying the core challenges of "where to interact" and "how to interact." Given only a single RGB-D image and a language instruction, the model predicts task-relevant functional region masks and 3D post-contact motion trajectories. A large-scale, standardized data pipeline integrates robotic manipulation, human demonstrations, simulation, and real-world scan data to construct a unified language–mask–3D motion affordance representation. Evaluated across eight benchmarks, the model substantially outperforms existing methods, achieving average gains of 23.9 in gIoU and 26.3 in cIoU, improving contact-point hit rates by 12.7–61.3%, and demonstrating superior 3D motion prediction. Notably, it enables zero-shot generalization across objects, tasks, and scenes and can be deployed directly on real robots without fine-tuning.

This work addresses the challenge of enabling robots to reliably plan and execute tasks from natural language instructions in complex, dynamic environments with occlusions, while achieving effective sim-to-real transfer. To this end, the authors propose a planning framework that integrates functional affordance recognition with visual action-effect prediction, leveraging visual forward reasoning to anticipate future states. A multimodal text-image matching module is introduced to evaluate the consistency between candidate action sequences and the linguistic goal. Furthermore, a real-to-sim image stylization mechanism is designed to enhance perceptual robustness in real-world settings. Experimental results demonstrate that the proposed approach successfully accomplishes challenging manipulation tasks on both simulated and physical robot platforms, significantly improving language-conditioned generalization from simulation to reality.