Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation

2026-07-23

AI Summary

This work addresses the limitation of existing robotic pretraining strategies that rely on salient cues in instructions rather than genuine semantic understanding, thereby hindering compositional generalization. The study formally introduces the concept of “instructional factor bias” and proposes two quantitative metrics—Factor Dominance Ratio (FDR) and Factor Dominance Hierarchy (FDH)—to diagnose distributional biases across factors such as color, object, and verb. Building upon this diagnostic framework, the authors devise a bias-aware data collection strategy that leverages instruction factor decomposition, data redistribution, and policy fine-tuning to optimize training data distributions. Experiments on both simulated and real-world robotic platforms demonstrate that the proposed approach achieves substantial improvements in sample efficiency and compositional generalization, significantly outperforming baseline methods using only half the demonstration data.

Abstract

Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual \textit{instruction factors}, \textit{e.g.,} reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, \textit{i.e.}, color $\geq$ object $\geq$ spatial $\geq$ verb $\geq$ size, with color dominant, and verb and size most under-grounded. We further show the diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget toward under-grounded factors outperforms baselines in simulation and on a real robot using half the demonstrations, thereby enabling more sample-efficient and generalizable policy learning.

Problem

• compositional generalization
• instruction grounding
• factor bias
• robotic manipulation
• language-conditioned policies

Innovation

• compositional generalization
• instruction factor bias
• bias-aware data collection
• Factor Dominance Rate (FDR)
• robotic manipulation