pseudo-label generation

This work investigates linear probing with a fixed feature extractor and reveals that, when conventional feature learning interpretations fail, self-distillation fundamentally performs label averaging guided by feature correlations encoded in the Gram matrix—yielding implicit label smoothing and enhanced noise robustness. We theoretically establish that under mild conditions, iterative self-distillation achieves 100% population accuracy. To avoid the computational overhead of multiple iterations, we propose PLL, an efficient single-round self-distillation paradigm: it constructs refined partial labels using the top-two softmax outputs from a teacher model. Leveraging spectral graph theory and deriving generalization error bounds, PLL significantly outperforms multi-round self-distillation on benchmarks including CIFAR-10/100, achieving state-of-the-art noise robustness—especially under high label noise—while maintaining theoretical rigor and practical efficiency.

Existing weak supervision evaluation methods typically rely on simplified tasks and balanced label distributions, failing to reflect real-world challenges such as severe class imbalance, domain-specific expertise, and multilingual parallelism. To address this gap, we propose BOXWRENCH—a novel benchmark and evaluation framework specifically designed for realistic constraints: expert-crafted labeling functions (LFs), multilingual support, high label cardinality, and long-tailed class distributions. BOXWRENCH integrates LF modeling, probabilistic label generation (e.g., Snorkel, FlyingSquid), and end-to-end pipelines, validated across multiple realistic NLP tasks. Empirically, weak supervision using only noisy, unlabeled data achieves performance comparable to—or even surpassing—that of fully supervised models trained on 1,000+ high-quality annotations. Our key contributions are (1) the first deployment-oriented weak supervision benchmark reflecting production constraints, and (2) empirical evidence demonstrating the practical viability of weak supervision in low-resource, high-complexity settings.

In programmatic weak supervision, manually designed labeling functions (LFs) are error-prone and heavily reliant on domain expertise. Method: This paper proposes an automatic LF repair method leveraging only 5–20 labeled examples. It models LFs as conditional rules and jointly optimizes their individual accuracy and discriminative evidential sufficiency for weak labels. Under a minimal-modification constraint, it employs a satisfiability-driven optimization framework to selectively refine LFs—adjusting only their trigger logic or output, without redesigning them from scratch. Contribution/Results: Evaluated on multiple benchmark tasks, the method significantly improves both LF accuracy and downstream model performance, demonstrating effective and robust LF repair under ultra-low annotation cost.

Noisy labels in large-scale web data degrade model performance, while existing methods lack reliable criteria for sample selection. To address this, we propose a surrogate-model-driven robust sample selection framework. Our approach replaces conventional static thresholds—based on loss or confidence—with a lightweight, transferable surrogate model, enabling generalization across diverse network architectures and datasets. The framework jointly optimizes sample confidence estimation through surrogate distillation, consistency regularization, dynamic threshold calibration, and a noise-robust loss function, all in an end-to-end manner. Extensive experiments on standard noisy-label benchmarks—including CIFAR-10, CIFAR-100, and WebVision—demonstrate that our method consistently outperforms state-of-the-art approaches such as FixMatch and Co-teaching, achieving absolute accuracy improvements of 3.2%–5.8%.

This work addresses the challenge of efficiently transferring knowledge from a teacher model to a student model in resource-constrained settings, particularly when the focus is on specific subtasks—such as minority classes and their associated concepts. To this end, the authors propose SubDistill, an algorithm that explicitly identifies and distills task-relevant subspace features at each layer of the network, enabling layer-wise, subtask-oriented knowledge transfer. By doing so, SubDistill enhances numerical stability and improves the fidelity of the student model to the teacher’s decision structure. Experimental results on multiple subtasks of CIFAR-100 and ImageNet demonstrate that SubDistill outperforms existing layer-wise distillation methods, with the student model more accurately replicating the teacher’s decision logic.

In retrieval-augmented search agents, feedback-enhanced self-distillation fails due to the model’s reliance on fixed output templates, leading to decoding collapse and generation of trajectories unrelated to the input. This work reveals that this phenomenon is overlooked by existing evaluation metrics and, for the first time, attributes the inconsistency in supervision signals to both the model architecture and prompting strategy. To address this, the authors propose integrating an exponential moving average (EMA) teacher model to stabilize the distillation signal, combined with KL-divergence-based supervision and trajectory analysis for optimization. Although the EMA teacher requires a warm-up phase, the approach ultimately yields significant performance gains on complex search tasks, demonstrating its effectiveness.

Weak supervision often fails to provide reliable signals for complex outputs, limiting the generalization and scalability of weak-to-strong model transfer. This work proposes a "weak-critic strong-supervision" paradigm, wherein a weak model acts as a non-misleading critic to guide a stronger model in more effectively leveraging its own knowledge. Through Online Progressive Critic Distillation (OPCD)—integrating weak-critic generation, high-quality critique filtering, an adaptive self-teacher mechanism, and alignment-aware training—the approach embeds high-fidelity criticism directly into the strong model’s optimization process. Experiments demonstrate consistent performance gains across reasoning and alignment benchmarks, offering a viable pathway toward scalable supervision using only weakly labeled data.

This study investigates the effectiveness boundaries of on-policy distillation in training reasoning models, with a focus on critical factors such as teacher selection, self-distillation context, and token-wise optimal policies. To this end, the authors propose a training-free diagnostic framework that, for the first time, quantifies at the token level the alignment between distillation signals and ideal gradient directions. This is achieved through a combination of ideal node gradient derivation and a scalable directional rollout algorithm to efficiently compute gradient alignment scores (measured via cosine similarity). The analysis reveals that distillation signals are more informative when the student errs but may introduce noise along correct reasoning paths. Furthermore, the optimal distillation configuration is highly dependent on both student capability and task characteristics, with no universally optimal setting.

This work addresses the challenge of learning in sparse-reward environments, where large language models struggle to acquire effective policies through environmental interaction, and existing self-distillation approaches overly rely on successful trajectories while neglecting the informative value of failure feedback. To overcome this limitation, the authors propose Reflection-Enhanced Self-Distillation (RESD), a novel framework that actively reconstructs failed trajectories into token-level supervision signals. RESD employs local reflection to diagnose errors and integrates a persistent global experience buffer to enable cross-episode knowledge reuse, thereby providing fine-grained supervision even in the complete absence of successful rollouts. Experimental results demonstrate that RESD significantly outperforms baseline methods across multiple continual learning tasks, achieving up to 8× higher sample efficiency than GRPO and markedly faster early-stage performance gains with only a single rollout per iteration.

This work addresses the challenge of leveraging consensus signals from large language models to improve reasoning accuracy in the absence of labeled data. The authors propose CANON, a novel method that, for the first time, transforms majority answers from multi-path reasoning into dense token-level supervision signals rather than merely using them for filtering or scalar rewards. CANON employs a frozen consensus-anchored teacher model to perform unlabeled self-distillation on the base model and introduces a consensus-guided conditional generation mechanism. Experimental results demonstrate that CANON achieves up to a 12-percentage-point improvement in pass@1 accuracy on mathematical and scientific reasoning tasks, outperforms unsupervised reinforcement learning by 6 points at roughly one-seventh the computational cost, approaches the performance of teacher models trained with gold labels, and successfully solves problems on which the original model failed across all 32 attempts, substantially enhancing majority-vote accuracy.