Generalizable Skill Learning for Construction Robots with Crowdsourced Natural Language Instructions, Composable Skills Standardization, and Large Language Model

2025-09-02

AI Summary

Construction robots suffer from poor skill generalization and high cross-scene reprogramming costs due to the quasi-repetitive nature of construction tasks. Method: This paper proposes a natural-language-instruction-driven general skill learning framework that innovatively integrates large language models (LLMs) with a modular skill standardization mechanism. It establishes a BIM-to-robot semantic data pipeline, enabling end-to-end mapping from natural language instructions to robot actions, and supports hierarchical multi-task skill transfer and compositional learning. Results: Evaluated on a full-scale industrial manipulator in long-duration drywall installation experiments, the framework demonstrates effective and practical multi-task reprogramming with minimal human intervention. It achieves high-quality task execution while significantly reducing programming overhead for deployment in new construction scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The quasi-repetitive nature of construction work and the resulting lack of generalizability in programming construction robots presents persistent challenges to the broad adoption of robots in the construction industry. Robots cannot achieve generalist capabilities as skills learnt from one domain cannot readily transfer to another work domain or be directly used to perform a different set of tasks. Human workers have to arduously reprogram their scene-understanding, path-planning, and manipulation components to enable the robots to perform alternate work tasks. The methods presented in this paper resolve a significant proportion of such reprogramming workload by proposing a generalizable learning architecture that directly teaches robots versatile task-performance skills through crowdsourced online natural language instructions. A Large Language Model (LLM), a standardized and modularized hierarchical modeling approach, and Building Information Modeling-Robot sematic data pipeline are developed to address the multi-task skill transfer problem. The proposed skill standardization scheme and LLM-based hierarchical skill learning framework were tested with a long-horizon drywall installation experiment using a full-scale industrial robotic manipulator. The resulting robot task learning scheme achieves multi-task reprogramming with minimal effort and high quality.

Problem

• Lack of generalizability in construction robot programming
• Inability to transfer skills across different work domains
• High reprogramming workload for alternate task performance

Innovation

• Crowdsourced natural language instructions for robot teaching
• Standardized hierarchical modeling for composable skills
• LLM-based framework enabling multi-task skill transfer