Score
Evaluating whether learned strategies, policies, or skills generalize to new tasks, scales, or environments (including zero-shot transfer) and diagnosing specialization and failure modes. Involves designing experiments and analyses to measure transfer, extract compact heuristics, and interpret when and why transfer succeeds or fails.
This study addresses the **reliable assessment of knowledge transferability** in transfer learning—a longstanding challenge hindered by inconsistent evaluation criteria, poor interpretability, and ill-defined applicability scopes. We propose the first **two-dimensional classification framework**, systematically organizing over 60 mainstream transferability metrics along axes of *transferable knowledge type* (e.g., features, relations, semantics) and *measurement granularity* (sample-, task-, or domain-level), while rigorously reconstructing their mathematical foundations, underlying assumptions, and failure boundaries. Through cross-modal and cross-task empirical analysis, we characterize the efficacy gradients and root limitations of metrics across paradigms (e.g., pretraining-finetuning). Our work establishes a standardized assessment pathway and principled metric selection guidelines for transferability evaluation, advancing trustworthy AI evaluation infrastructure, and identifying key future directions—including dynamic transferability modeling and causally grounded metrics.
This work addresses the low sample efficiency and poor generalization of training robotic skills from scratch, focusing on the plug-insertion task. It systematically evaluates policy transfer across different robot platforms by comparing zero-shot transfer, fine-tuning, and training from scratch. The authors propose a policy transfer framework incorporating adaptive mechanisms that significantly enhance cross-platform generalization without requiring extensive retraining. Experimental results demonstrate that fine-tuning with only a small amount of interaction data substantially outperforms both zero-shot transfer and training from scratch, achieving state-of-the-art performance in terms of both success rate and execution efficiency. This approach offers a promising pathway toward sustainable and data-efficient robot learning.
This study investigates the intrinsic mechanisms underlying zero-shot generalization in instruction tuning, revealing that it fundamentally arises from instance-level input similarity—termed “early generalization”—and is highly sensitive to training data ordering. To address this, we propose the first test-centric multi-turn arrangement framework, which integrates dynamic loss analysis, instance similarity modeling, granularity-aware sorting, and progressive training scheduling. Our method significantly enhances zero-shot generalization on unseen tasks, accelerates convergence, reduces training loss, and improves generalization stability. The core contribution is the first empirical identification of instance-level similarity—not task-level structural alignment—as the primary driver of zero-shot generalization; further, we establish data ordering as a novel, controllable lever for optimizing generalization behavior, enabling precise, target-driven adaptation without architectural or objective modifications.
This work addresses the long-standing isolation among research domains such as alignment training, model organisms, and toy models, which has hindered empirical cross-pollination and led to redundant exploration and inefficiency. For the first time, it systematically transfers supervised fine-tuning (SFT) practices across these domains by integrating cross-model output training, mixed-strategy data, and benign fine-tuning to rigorously evaluate the portability of key findings. The study demonstrates three successful transfer effects: enhanced behavioral generalization, mitigation of capability degradation, and the critical insight that preserving capabilities alone is insufficient to ensure robustness in subsequent training phases. These results underscore both the efficacy and limitations of reusing methodologies across domains, thereby fostering more synergistic development across disparate research areas.
Addressing the fundamental limitations of artificial neural networks—namely, poor systematic generalization, catastrophic forgetting, few-shot learning inefficiency, and inadequate multi-step reasoning due to the absence of human-like cognitive development mechanisms—this paper proposes a meta-learning framework explicitly optimizing for “motivation + practice.” Unlike conventional paradigms driven by indirect objectives (e.g., loss minimization), our framework integrates differentiable optimization, curriculum learning, task embedding, and practice trajectory modeling to enable models to autonomously acquire skill-improvement motivation and structured training opportunities during learning. Evaluated across four benchmark task families, it significantly outperforms state-of-the-art methods, demonstrating the efficacy of the motivation-practice mechanism for robust generalization and continual learning. Moreover, it establishes, for the first time, a computationally tractable cognitive development pathway for neural networks—bridging machine learning and cognitive science through a novel, principled paradigm.
This work investigates the mechanisms underlying generalization degradation in multi-task learning (MTL), specifically why mainstream optimization algorithms often fail to improve generalization. Through controlled experiments on deep neural networks, quantitative measurement of gradient conflicts, visualization of optimization trajectories, and attribution analysis of generalization error, we empirically establish— for the first time—that a substantial generalization gap emerges early in multi-task training, and crucially, this gap is decoupled from the degree of gradient conflict: while gradient conflicts shape task-specific optimization dynamics, they do not predict generalization performance. This finding challenges the prevailing MTL design paradigm that assumes mitigating gradient conflict is a necessary condition for improving generalization. Our primary contributions are: (i) identifying the early onset and independence of MTL generalization failure; (ii) refuting gradient conflict as a valid proxy for generalization; and (iii) providing new theoretical foundations for understanding task synergy and developing robust multi-task optimizers.
Current evaluation methods focus primarily on overall model performance on i.i.d. test sets, failing to characterize the transfer scope of knowledge acquired from individual training examples. This work proposes the “generalization spectrum”—a chromatography-inspired evaluation framework that formalizes human-like analogical learning into a measurable dimension by constructing, for each training sample, a sequence of test variants ordered by increasing transfer distance (e.g., exact recall, cross-lingual implementation, narrative reconstruction). Leveraging controlled variant design and selection-based synthetic data generation, we compare multiple learning paradigms on programming tasks and find that reinforcement learning supports near-transfer more effectively than supervised fine-tuning, whereas self-distillation or prompt-augmented approaches may impair far-transfer capabilities, revealing a nontrivial trade-off between learning depth and transfer breadth.
This work addresses systematic limitations in existing creative quality alignment (CQA) datasets, particularly their inadequate modeling of audience preferences and insufficient coverage of real-world logical constraints. To overcome these issues under stringent engineering and data scarcity conditions, the authors propose a low-resource CQA approach that leverages only around one hundred expert-annotated chain-of-thought (CoT) examples. By uncovering a dual mechanism between appreciation and generation tasks within conditional generative architectures, the method enables automatic transfer of calibrated knowledge from the appreciation module to the generation module. Experimental results demonstrate that the proposed framework substantially mitigates the shortcomings of current datasets and validates the practical feasibility of aligning generative models with nuanced creative quality metrics in real-world engineering settings.
Existing research lacks a systematic understanding of the full lifecycle of model-generated skills—spanning experience generation, skill extraction, and skill consumption—making it difficult to evaluate their effectiveness and applicability. This work proposes the first utility-based evaluation framework to systematically analyze key factors influencing skill extraction and consumption across five task domains. Through multi-model comparisons, cross-consumer transfer tests, and analyses of experience composition, we find that extracted skills are on average beneficial but exhibit significant negative transfer, with utility independent of model scale. Furthermore, we introduce a meta-skill guidance strategy that substantially improves cross-domain skill quality and mitigates negative transfer, revealing a notable inconsistency between extractor and consumer performance.
This study addresses the inefficiency in mastery learning caused by learners’ suboptimal task-selection strategies—such as challenge avoidance—and their interaction with system-imposed algorithmic constraints. For the first time, it integrates empirically grounded models of real student strategic behavior into a mastery learning simulation framework. Leveraging behavioral models derived from authentic interaction data, the work simulates distinct learning strategies (e.g., weakness-focused practice and interleaving) and employs over-practice metrics to quantify how system constraints modulate inefficient behaviors. Findings reveal that risk-averse strategies substantially exacerbate over-practice in multi-step complex tasks, yet targeted system constraints effectively mitigate such inefficiencies without compromising the performance of more effective strategies. This approach offers a low-cost validation method for intelligent tutoring systems prior to deployment.