self-supervised learning

While contemporary self-supervised and masked/denoising autoencoder methods effectively learn strong representations from massive unlabeled data, their representational nature, cross-task generalization capability, and emergence mechanisms remain theoretically unexplained. Method: This project integrates statistical inference and nonconvex optimization theory to establish a unified analytical framework for unsupervised representation learning. Contribution/Results: It provides the first mathematical characterization of how self-supervised objectives—such as contrastive learning and reconstruction losses—induce structured latent spaces, and quantitatively links representation linear separability, invariance, and downstream generalization. The work identifies key theoretical conditions under which pretrained models achieve zero-shot transfer and task emergence in vision foundation models. Crucially, it delivers the first theoretical foundation for large-scale pretraining that is both statistically interpretable and optimization-traceable—bridging statistical guarantees with practical training dynamics.

This work investigates whether supervised learning models can be trained and generalized without access to any ground-truth labels $y$. To this end, we propose the *$y$-free smooth operator* paradigm, which constructs models as smooth mappings $S(x)$ dependent solely on input features $x$, enabling fully label-free training. We provide the first theoretical guarantee that supervised models can achieve effective training without ground-truth labels. Furthermore, we introduce an unsupervised model selection criterion based on predictive distribution consistency, circumventing the conventional reliance on labeled data for cross-validation. Empirical evaluation on synthetic and real-world datasets demonstrates that linear/kernel ridge regression, spline smoothing, and neural networks—trained exclusively on random (i.e., meaningless) labels—achieve performance comparable to standard supervised learning and substantially surpass random guessing. These results empirically validate the core finding: ground-truth labels are not strictly necessary for effective supervised model training and generalization.

To address out-of-distribution (OOD) generalization failure caused by spurious correlations and feature noise, this paper proposes a two-stage representation learning paradigm: first freezing core features, then training remaining parameters via empirical risk minimization (ERM). Theoretically, we establish for the first time that non-realizable noise fundamentally constrains the learnability of core features; leveraging this insight, we design Feature-Freezing and Tuning (FTT), ensuring test-time linear probes robustly rely on decoupled core representations. Our method integrates feature freezing, linear-probe-based diagnostic evaluation, and provably guaranteed representation disentanglement analysis. Experiments demonstrate a 4.5% accuracy gain over strong baselines—including ERM, IRM, JTT, and CVaR-DRO—on high-noise spurious-correlation benchmarks. Moreover, FTT achieves superior OOD generalization across standard distribution shift benchmarks, validating its effectiveness and robustness.

This work addresses the challenge of enabling generative language models to selectively forget sensitive data under privacy regulations such as GDPR—without access to the original training data. To this end, we propose Iterative Contrastive Unlearning (ICU), the first framework that integrates knowledge-guided unlearning, contrastive learning augmentation, and dynamic evaluation optimization. ICU employs a lightweight iterative fine-tuning paradigm, jointly optimizing a knowledge-unlearning loss and a contrastive objective to balance forgetting efficacy and model utility. Crucially, it operates in a data-free setting. Extensive experiments across multiple benchmarks demonstrate that ICU achieves over 98% removal of sensitive information while preserving more than 95% of the original language modeling performance—substantially outperforming state-of-the-art unlearning methods. ICU thus provides a scalable, data-free, and regulation-compliant solution for privacy-preserving large language model deployment.

This work addresses the false sense of data privacy induced by unlearnable examples—samples perturbed to cause premature overfitting to spurious features while suppressing semantic learning. We propose a progressive, multi-stage training paradigm grounded in our novel observation that models initially learn both perturbation and semantic features, yet shallow layers rapidly overfit to perturbations. To break this unlearnability bottleneck, we design a dynamic hierarchical freezing/unfreezing mechanism. Our method integrates progressive layered network training, adaptive parameter scheduling, and multi-stage loss formulation, and is compatible with mainstream architectures including CNNs, ResNets, and Vision Transformers (ViTs). Extensive experiments on CIFAR-10/100 and ImageNet-mini demonstrate substantial improvements over existing defenses, establishing our approach as a new benchmark for evaluating unlearnability mitigation techniques.

This study investigates whether task-agnostic structural proxies reliably explain variations in out-of-distribution (OOD) transfer performance of pretrained models. Under fixed pretraining and probing conditions, the authors construct the first controlled counterexamples—through theoretical reasoning and synthetic sequence modeling experiments—that disentangle structural proxies from task-relevant structure. By integrating formal structural measures (e.g., epiplexity), operational proxy metrics, synthetic data, and multi-seed probing evaluations, they demonstrate a reversal in the ranking between structural proxies and OOD accuracy across two random seeds. This finding exposes fundamental limitations of structural proxies in analyzing OOD generalization and challenges their validity as universal explanatory tools for transfer performance.

This work addresses the challenge of data-scarce real-world scenarios by proposing an efficient label-free meta-learning framework that eliminates the need for computationally expensive model inversion used in existing data-free approaches. Instead, the method leverages a pre-trained model to generate soft labels for unlabeled data, constructing meta-tasks without synthetic data generation. To enhance meta-training efficacy, it introduces a task-weighting mechanism based on task confidence and class distribution balance. The proposed approach achieves substantial improvements in both efficiency and performance, yielding accuracy gains of 8.4%–36.4% on few-shot classification benchmarks while accelerating computation by up to 104× compared to prior methods.

This work addresses the challenge that privileged information—available during training but inaccessible at deployment—can mislead models when it is noisy or weakly informative. To mitigate this issue, the authors propose a joint training framework that simultaneously optimizes a teacher model leveraging privileged information and a student model restricted to inputs available at test time. Through an innovative coupling mechanism and an alternating optimization algorithm, the student selectively distills useful knowledge from the teacher while avoiding the propagation of its errors. Theoretical analysis establishes conditions under which this joint training improves accuracy and supports efficient implementation even for high-dimensional, large-scale models. Experiments on both synthetic and real-world datasets demonstrate that the proposed method significantly outperforms conventional two-stage baselines and exhibits robustness to low-quality privileged information.

This work addresses semi-supervised regression in settings where labeled data are scarce but abundant unlabeled data are available alongside noisy pre-trained representations serving as proxy covariates. The authors propose a two-stage estimator that first leverages all proxy covariates to learn kernel features and then fits a ridge regression predictor using the limited labeled samples. Under conditions of controllable proxy noise and sufficient unlabeled data, the method recovers the fast convergence rate dictated by the number of labeled samples. The theoretical analysis establishes finite-sample generalization bounds and unifies distribution regression within the same framework, providing corresponding performance guarantees. Empirical results demonstrate that the proposed approach significantly outperforms existing supervised and semi-supervised baselines, particularly in low-label regimes.

This work investigates the optimal choice of representation dimensionality to achieve maximal generalization performance in settings where pretraining and downstream task data exhibit significant imbalance. By modeling pretraining as principal component analysis on unlabeled data and downstream learning as linear regression on labeled data, the authors employ high-dimensional statistical analysis to derive, for the first time, exact expressions for training and generalization errors in the high-dimensional asymptotic limit. Their theoretical results establish a quantitative relationship among representation dimensionality, data scale, and task alignment, yielding an analytical condition for the optimal representation size: low-dimensional compressed representations are preferable when labeled downstream data are scarce but pretraining data are abundant, whereas high-dimensional representations are advantageous in the opposite regime. The study further quantifies the extent to which unlabeled data can substitute for labeled samples and validates the proposed mechanism in both autoencoders and large language models.