Let the Target Select for Itself: Data Selection via Target-Aligned Paths

2026-05-10

AI Summary

Existing data selection methods introduce bias in heterogeneous data pools due to their reliance on reference trajectories misaligned with downstream task objectives. This work proposes a zeroth-order data selection approach that leverages a compact warm-up trajectory induced by the target validation set as an aligned and decoupled reference path, scoring candidate samples via normalized loss reduction at the trajectory endpoint. The method requires neither gradients nor Hessian approximations and enables reuse of the warm-up trajectory across diverse data pools, substantially reducing computational and storage overhead. Empirical evaluations across logistic regression, vision, and instruction fine-tuning tasks demonstrate performance comparable to strong dynamic attribution baselines while significantly decreasing warm-up time and storage costs.

Abstract

Targeted data selection aims to identify training samples from a large candidate pool that improve performance on a specific downstream task. Many recent methods estimate candidate utility by aggregating local attribution scores along a trajectory induced by the candidate pool. When the pool is heterogeneous, however, this reference trajectory may be misaligned with the dynamics of a target-aligned selected subset, creating what we call reference path bias. We propose an alternative reference path: a validation-induced flow obtained from a short, capacity-limited warmup on the available target validation proxy. Along this path, candidates are scored by a normalized endpoint loss drop, yielding a simple zero-order selection rule that requires no candidate gradients or Hessian approximations. Across controlled logistic, vision, and instruction-tuning experiments, this score is competitive with strong dynamic attribution baselines while substantially reducing warmup and storage cost. Moreover, since the reference trajectory is decoupled from any specific candidate pool, the same compact warmup can be reused across additional pools without recomputing the trajectory.

Problem

• data selection
• targeted data selection
• reference path bias
• heterogeneous data
• downstream task

Innovation

• target-aligned data selection
• reference path bias
• validation-induced flow
• zero-order selection
• data efficiency