Diagnosing Compositional Generalization in Sequential Robot Tasks

2026-07-31

AI Summary

This work addresses the failure of robotic agents to generalize to unseen instruction combinations due to sparse training data by proposing a tripartite decomposition framework of the compositional generalization gap. The gap is attributed to marginal instruction shift, instruction-combination shift, and context-action shift. Through instruction-space coverage analysis and structured subset sampling, the study demonstrates that structural coverage of instruction dependencies is more critical than exhaustive task enumeration and identifies the generalization bottleneck as stemming from instruction-guidance bias rather than a lack of underlying skills. Remarkably, strong out-of-distribution (OOD) generalization is achieved using only a quarter of the full task space via a carefully structured subset; when combined with single-example fine-tuning, OOD task success rates improve dramatically from 0.4% to 54.7%.

Abstract

Sequential robot manipulation requires policies to execute novel combinations of familiar instruction components. However, collecting demonstrations for all possible instruction tuples is combinatorially expensive, while sparsely covered datasets often fail under out-of-distribution recombination. This paper studies compositional generalization through the lens of instruction-space coverage. We decompose the generalization gap into three sources: \textit{marginal instruction shift}, \textit{instruction-compositional shift}, and \textit{context--action shift}. This decomposition allows us to diagnose when sparse training coverage is sufficient, and what structure the training set must preserve for reliable action prediction. Our results show that exhaustive tuple enumeration is unnecessary: a structured subset, as small as one quarter of the full task space, can recover strong out-of-distribution performance when it covers action-relevant dependencies. We further find that sparse training often fails due to instruction steering rather than missing low-level skills; finetuning only one demonstration per task improves OOD success from \(0.4\%\) to \(54.7\%\). For semantically dependent tasks, effective coverage must capture relational structure rather than only factor diversity. These findings suggest that efficient robot data collection should prioritize dependency coverage in instruction space over exhaustive task expansion. More results are available in the supplementary material. Project website: https://yixiaowang7.github.io/Diagnosing_Compositional_Generalization_Robot_Page/.

Problem

• compositional generalization
• instruction-space coverage
• out-of-distribution recombination
• sequential robot tasks
• data efficiency

Innovation

• compositional generalization
• instruction-space coverage
• dependency structure
• out-of-distribution generalization
• robot manipulation