Assessing VLM-Driven Semantic-Affordance Inference for Non-Humanoid Robot Morphologies

2026-04-21

AI Summary

This work addresses the lack of systematic evaluation of vision-language models (VLMs) for semantic affordance reasoning in non-anthropomorphic robotic systems by introducing the first comprehensive benchmarking framework. Leveraging a hybrid dataset that combines real-world annotations with VLM-generated synthetic scenes, the study thoroughly assesses VLM generalization across diverse robot morphologies and object categories. Findings reveal that while VLMs exhibit notable generalization potential, they consistently display a conservative prediction bias—characterized by low false-positive rates but high false-negative rates—particularly pronounced in novel tool-use and unconventional manipulation scenarios. These results underscore the necessity of integrating complementary approaches to balance safety and recall, offering new directions for affordance perception in heterogeneous robotic systems.

Technology Category

• Computer Vision: Language and Vision
• Intelligent Robots: Manipulation
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding human-object interactions, but their application to robotic systems with non-humanoid morphologies remains largely unexplored. This work investigates whether VLMs can effectively infer affordances for robots with fundamentally different embodiments than humans, addressing a critical gap in the deployment of these models for diverse robotic applications. We introduce a novel hybrid dataset that combines annotated real-world robotic affordance-object relations with VLM-generated synthetic scenarios, and perform an empirical analysis of VLM performance across multiple object categories and robot morphologies, revealing significant variations in affordance inference capabilities. Our experiments demonstrate that while VLMs show promising generalisation to non-humanoid robot forms, their performance is notably inconsistent across different object domains. Critically, we identify a consistent pattern of low false positive rates but high false negative rates across all morphologies and object categories, indicating that VLMs tend toward conservative affordance predictions. Our analysis reveals that this pattern is particularly pronounced for novel tool use scenarios and unconventional object manipulations, suggesting that effective integration of VLMs in robotic systems requires complementary approaches to mitigate over-conservative behaviour while preserving the inherent safety benefits of low false positive rates.

Problem

• vision-language models
• affordance inference
• non-humanoid robots
• robotic embodiment
• semantic affordance

Innovation

• vision-language models
• affordance inference
• non-humanoid robots
• hybrid dataset
• conservative prediction