🤖 AI Summary
Existing demonstration filtering metrics struggle to identify structural flaws that degrade imitation learning performance, particularly failing when relying solely on action information. This work constructs a controlled testbed by injecting known defects—such as action noise, tremor, truncation, and critical-step errors—to systematically evaluate the effectiveness of seven filtering metrics in detecting such flaws and improving downstream policy performance. The study reveals, for the first time, that action-only metrics are not only ineffective but actively harmful in the presence of structural errors, while state-aware metrics partially mitigate these issues yet recover at most one-third of the lost performance. Notably, high defect detection accuracy does not necessarily translate into policy gains. The authors open-source their testbed and metric implementations, underscoring the necessity of state-trajectory analysis in demonstration assessment.
📝 Abstract
Imitation-learning policies inherit the quality of the demonstrations they are trained on, and a growing set of curation metrics promise to score and filter low-quality demonstrations automatically. These metrics are each validated on different data with different protocols, so it is unclear which of them actually identify the demonstrations that harm a policy. We build a controlled testbed in which demonstration defects are injected with known type, and audit seven curation metrics along two axes: how well each separates defective from clean demonstrations, and whether training a behavior-cloning policy on each metric's curated subset improves task success. We study two defect regimes. Subtle perturbations (correlated action noise, tremor, truncation) are detectable by multivariate outlier scoring and, once removed, recover the full downstream gap. Structural errors, where the demonstration executes a wrong action at a key moment, are invisible to every action-only metric we test, and two of them are inverted: they score defective demonstrations as higher quality and, used for curation, tend to leave the policy at or below the uncurated baseline rather than above it. Only metrics that examine the state trajectory detect structural errors, and even the best of them recovers just a third of the downstream gap. High detection accuracy does not guarantee downstream improvement. We release the testbed and all curation implementations.