Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends

2026-06-10

AI Summary

This work addresses the high complexity of benchmark construction in embodied intelligence evaluation, which entails multidimensional challenges including task definition, environment setup, data annotation, and metric design. The authors propose a five-stage automated pipeline that integrates conventional automation, foundation model assistance, and agent-in-the-loop workflows to systematically unify task design, data acquisition, cleaning and annotation, benchmark generation, and evaluation feedback. The study demonstrates that automation’s core value extends beyond cost reduction to enhancing auditability, version control, and long-term governance, thereby enabling a diagnostic, traceable, and sustainably evolvable evaluation framework. A systematic analysis of cost structures across pipeline stages further provides methodological guidance for future embodied intelligence benchmarks that balance scalability with responsible automation.

Abstract

Embodied intelligence now spans navigation, household assistance, manipulation, autonomous driving, aerial agents, and multimodal large-model control. This expansion has made benchmark construction a central bottleneck for reliable evaluation. Unlike static datasets, embodied benchmarks combine task specifications, environments, robot data, demonstrations, annotations, metrics, evaluation scripts, and release policies into a single evaluation system. This survey reviews the literature through a five-stage construction pipeline: requirement and task construction, data acquisition, data cleaning and annotation, benchmark suite generation and metric definition, and evaluation execution with diagnostic feedback. For each stage, the survey analyzes the transition from manual curation to traditional automation, foundation-model assistance, and agentic closed-loop workflows. It also compares qualitative construction costs across human labor, data and asset acquisition, compute and simulation, validation and debugging, governance and maintenance, and rework risk. The main conclusion is that automation does not simply reduce benchmark cost. Instead, it often shifts cost toward validation, auditability, version control, and long-term governance. Progress in embodied evaluation will therefore depend not only on larger benchmark suites, but also on construction pipelines that are diagnosable, auditable, and responsibly refreshable.

Problem

• embodied intelligence
• benchmark construction
• evaluation systems
• automation
• benchmarking bottleneck

Innovation

• embodied intelligence
• benchmark construction
• intelligent automation
• foundation models
• evaluation pipelines