Affordance-Based Manipulation Planning with Text Goals and Sim-to-Real Generalisation via Real-to-Sim Image Conversion

2026-07-12

AI Summary

This work addresses the challenge of enabling robots to reliably plan and execute tasks from natural language instructions in complex, dynamic environments with occlusions, while achieving effective sim-to-real transfer. To this end, the authors propose a planning framework that integrates functional affordance recognition with visual action-effect prediction, leveraging visual forward reasoning to anticipate future states. A multimodal text-image matching module is introduced to evaluate the consistency between candidate action sequences and the linguistic goal. Furthermore, a real-to-sim image stylization mechanism is designed to enhance perceptual robustness in real-world settings. Experimental results demonstrate that the proposed approach successfully accomplishes challenging manipulation tasks on both simulated and physical robot platforms, significantly improving language-conditioned generalization from simulation to reality.

Abstract

We present a manipulation planning system based on affordance recognition and action effect prediction. The system reasons through possible futures in visual form, and evaluates candidate plans by agreement of predicted outcomes with text-based goals set at run-time, using a multi-modal goal-matching module. Positions of objects named in the goal text are tracked through predictions even when occluded, making it possible to generate action plans even when objects become occluded, or when their initial descriptors cease to identify them in future states. We further expand the system with an image conversion module for translating real-world state images with objects of varied shapes and visual appearances into a consistent visual appearance, to facilitate manipulation planning in a physical robot setup. We evaluate performance of the system's modules in isolation and demonstrate the integrated system's manipulation planning capabilities on a set of challenging tasks in both simulation and on hardware.

Problem

• affordance-based manipulation
• text goals
• sim-to-real generalisation
• occlusion handling
• real-to-sim conversion

Innovation

• affordance-based manipulation
• text-goal planning
• sim-to-real generalization
• real-to-sim image conversion
• occlusion-robust tracking