cross-task transfer analysis

This work addresses the fundamental lack of fairness and robustness in evaluating model transferability across domains. We propose the first systematic, standardized benchmarking framework for assessing cross-domain transfer capability. Our method introduces a unified multi-source-domain–target-domain evaluation protocol, encompassing diverse transfer tasks and perturbation-robustness analysis, and adopts head-training (i.e., linear-probe fine-tuning) as the consistent evaluation paradigm. Empirical analysis reveals significant performance discrepancies among existing transferability metrics under varying experimental settings, undermining their reliability. Our framework substantially improves assessment fidelity, yielding an average 3.5% gain in transfer performance under standard head-training configurations. To foster reproducibility and rigorous comparison, we fully open-source all code, datasets, and evaluation pipelines—establishing a new, standardized paradigm for transferability measurement.

This study investigates whether improvements in AI models’ reasoning capabilities naturally enhance their effectiveness as teachers—specifically, their ability to convey understandable and transferable knowledge to humans in human-AI collaboration. Method: We propose KITE, a novel evaluation framework that quantifies the causal impact of model explanations on humans’ subsequent independent problem-solving performance via a two-stage behavioral experiment (N=118). Contribution/Results: We provide the first systematic definition and empirical measurement of human-AI knowledge transfer efficacy, revealing only weak correlation between standard model benchmark performance and actual knowledge transfer—evidencing a “knowledge-rich but explanation-poor” phenomenon. We identify key behavioral strategy factors governing successful knowledge transfer and demonstrate that explainability and pedagogical alignment must be explicitly optimized. To support reproducibility and further research, we open-source the KITE toolkit—including implementation code, annotated datasets, and standardized evaluation protocols.

This study identifies the long-standing neglect of validity threats—particularly carryover effects—in crossover-design experiments within software engineering (SE). Method: Following Vegas et al.’s guidelines, we conducted the first quantitative assessment of 67 crossover-design experiments reported in 136 SE papers (2015–2024), employing forward snowball sampling and systematic content coding. Contribution/Results: Only 29.5% of validity threats were adequately addressed, and carryover effects were explicitly modeled in a mere 3% of studies. Although overall analytical validity has improved compared to earlier periods, practical adherence remains critically insufficient. Crucially, this work provides the first empirical evidence that low guideline adoption is a primary bottleneck. We propose a novel, taxonomy-based framework for classifying and assessing validity threats, offering actionable pathways and empirical grounding to enhance methodological rigor in SE experimentation.

This work addresses the limited goal-directed execution capability of large language models in long-horizon tasks by introducing a Goal-Directed Execution (GDE) behavioral framework. The authors conduct post-training on the Qwen3.5-122B-A10B model using 363 long-horizon, multi-tool agent tasks from office scenarios, without relying on software engineering data. This approach yields a notable improvement on SWE-Bench Pro, increasing pass@1 by 5.8 percentage points. Experimental results demonstrate significant enhancements across four core GDE capabilities: goal selection, state construction, goal consistency maintenance, and environment validation. Furthermore, the model exhibits effective cross-domain transfer between office and software engineering tasks, confirming that long-horizon post-training can successfully drive the transfer of behavioral mechanisms.

Addressing the challenge of reliably inferring AI systems’ cognitive capabilities from heterogeneous, few-shot task performance, this paper proposes a Bayesian triangulation framework for cognitive profiling. The method introduces a “measurement layout” generative model (implemented in PyMC) that jointly models task-instance features, latent capability dimensions, and system responses—thereby overcoming traditional psychometric reliance on large-scale, homogeneous datasets. Its key innovation lies in the first integration of Bayesian latent-variable modeling with multi-task cross-validation, enabling individualized, architecture-agnostic cognitive capability inversion. Evaluated on the AnimalAI Olympics benchmark (68 competing agents) and the O-PIAAGETS benchmark (30 synthetic agents), the framework successfully reconstructs fine-grained cognitive profiles, significantly enhancing discriminability and interpretability of inferred capabilities. Results empirically validate the feasibility and effectiveness of capability-oriented evaluation as a principled alternative to conventional behavioral benchmarks.

研究通过对比任务级与子任务级技能诱导及文本与代码格式，解决LLM代理技能转移不可靠问题，提出技能效用评分以预测任务成功。

This study addresses the inference bias that arises when organizations evaluate expert competence solely based on project success or failure, a distortion attributable to differences in task bundling architectures. Drawing on Bayesian inference and Blackwell’s partial order theory, this work compares the informational value of bundled, outsourced, and unbundled projects, derives reliability thresholds, and quantifies the statistical costs associated with coarse-grained aggregation. The primary contribution lies in establishing, for the first time, a precise threshold relationship between task architecture and learning efficiency, revealing that under fixed workloads, the advantage of bundling strengthens as task scope expands. Furthermore, it demonstrates that bundling dominates when external technologies are unreliable, and that interim auditing can significantly broaden its region of superiority.

This work addresses systematic limitations in existing creative quality alignment (CQA) datasets, particularly their inadequate modeling of audience preferences and insufficient coverage of real-world logical constraints. To overcome these issues under stringent engineering and data scarcity conditions, the authors propose a low-resource CQA approach that leverages only around one hundred expert-annotated chain-of-thought (CoT) examples. By uncovering a dual mechanism between appreciation and generation tasks within conditional generative architectures, the method enables automatic transfer of calibrated knowledge from the appreciation module to the generation module. Experimental results demonstrate that the proposed framework substantially mitigates the shortcomings of current datasets and validates the practical feasibility of aligning generative models with nuanced creative quality metrics in real-world engineering settings.

论文针对代理基准测试中的双重测量混淆问题，通过将关键决策转移给模型、使用基于真实值的评分及报告更全面的可靠性指标来解决。

This study addresses the current lack of human-centered, interpretable, and responsible evaluation criteria for AI in modeling and simulation. The authors propose the first multidimensional benchmark framework specifically designed to assess large language models (LLMs) through a human-centric lens, leveraging an open-source system dynamics AI platform to systematically evaluate performance across qualitative modeling, quantitative modeling, and model discussion tasks—emphasizing human-AI collaboration rather than replacement. The framework incorporates critical capabilities such as causal reasoning, iterative model refinement, and behavioral explanation, while embedding ethical and accountability considerations. Empirical results indicate that existing AI tools perform relatively well in qualitative tasks and model discussions but remain limited in causal reasoning and quantitative error correction; furthermore, different LLMs exhibit distinct strengths, with no single model emerging as universally superior.