The Composition Gap in Dataset Distillation

2026-09-28

AI Summary

This study addresses the significant performance degradation observed when merging multi-source independently distilled datasets in federated learning. By integrating dataset distillation, Hessian variance analysis, and trajectory compression theory, this work derives, for the first time, an exact composition error formula under quadratic objectives, decoupling the error into local bias and residual components. The analysis reveals fundamental nonlinear discrepancies between independent and joint distillation, demonstrating that training fidelity and downstream accuracy requirements are inherently distinct. Furthermore, it proves that single-dataset evaluation cannot guarantee joint performance and identifies that residuals dominate the error under endpoint matching conditions. These findings provide rigorous theoretical justification for the advantages of joint distillation in federated settings.

Abstract

Dataset distillation compresses a training set into a small synthetic set, usually evaluated one at a time. In federated and data-governance settings, several parties distill their own data and a user trains on their union. We ask whether the union of separately distilled sets reproduces training on the union of the real data composability and show that it can fail even when every source is distilled exactly and the total budget admits an exact joint distillate. Compressing a training trajectory into fewer steps transforms the source statistics nonlinearly, so averaging compressed sources differs from compressing their average. For quadratic objectives we derive the exact composition error for two-to-one step compression in terms of the source-Hessian variance and the linear terms of the losses, and on a smooth network at small step sizes this prediction captures the local endpoint discrepancy in magnitude and direction. For learned synthetic sets, however, the composed error decomposes exactly into this local discrepancy and an aggregate source residual. Under endpoint matching the residual exceeds the structural term by more than an order of magnitude, and under distribution matching the two terms partly cancel. Joint distillation also retains an accuracy advantage when both sets are distilled from the same dataset, where the local discrepancy is exactly zero. Training fidelity and downstream accuracy are therefore distinct requirements, neither established by evaluating each set on its own.

Problem

• Dataset Distillation
• Composability
• Composition Gap
• Federated Learning

Innovation

• Dataset Distillation
• Composition Gap
• Composability
• Federated Learning
• Trajectory Compression