knowledge consolidation

This study addresses the challenge of heterogeneous knowledge transfer—i.e., effective knowledge propagation across models with incompatible architectures, tasks, and data modalities. We propose a general-purpose transfer framework that requires neither structural alignment nor label-space compatibility. Our core innovation is a parameter-space bridging mechanism: a learnable low-rank adapter dynamically extracts knowledge from the source model’s training trajectory and adaptively maps it into the target model’s parameter space. By integrating low-rank parameter querying with end-to-end differentiable joint training, the framework enables seamless cross-modal, cross-task, and cross-architecture optimization. Experiments across multiple highly heterogeneous transfer benchmarks demonstrate substantial improvements over state-of-the-art methods. Notably, our approach remains robust and efficient even in extreme heterogeneity scenarios where conventional approaches fail completely.

Addressing key challenges—including difficulty in cross-capability transfer among large-scale heterogeneous language models, catastrophic forgetting in knowledge distillation, and insufficient knowledge absorption in parameter-efficient fine-tuning (PEFT)—this paper proposes GraftLLM, a novel framework for capability reuse. Methodologically, it introduces: (1) SkillPack, a structured knowledge carrier that decouples and encapsulates source-model capabilities into modular skill units; (2) a module-aware adaptive parameter compression strategy to mitigate parameter interference while preserving the target model’s intrinsic capacity; and (3) a lightweight, full-parameter-free knowledge grafting mechanism enabling forget-free continual learning. Evaluated on multi-task transfer, model fusion, and continual learning benchmarks, GraftLLM substantially outperforms baselines such as FuseLLM—achieving a 23.6% improvement in knowledge retention, enhanced generalization, and a 41% reduction in computational overhead. The framework establishes a scalable, low-cost paradigm for large language model capability reuse.

To address the challenges of fine-grained knowledge extraction, inefficient aggregation, and suboptimal accuracy in multi-source transfer learning, this paper proposes a lightweight knowledge fusion framework based on Singular Value Decomposition (SVD). Methodologically, each source model is decomposed layer-wise into rank-one components; salient components are selected according to significance, and only the principal singular values of the fused matrix are fine-tuned for target-task adaptation. This mechanism achieves both high accuracy and efficiency: it avoids full-parameter fine-tuning, drastically reducing retraining overhead; exhibits robustness to noise and pruning-induced perturbations; and scales effectively to large, high-parameter models. Experiments demonstrate substantial performance gains across diverse multi-source transfer tasks, strong computational scalability, and establish a novel paradigm for efficient model knowledge reuse.

Model merging often suffers from unpredictable performance, limiting its practical utility. This work introduces the first quantifiable definition of model mergeability and systematically investigates the key factors influencing merging effectiveness, identifying the base model’s prior knowledge about the fine-tuning data as the decisive factor. Building on this insight, the authors propose a weighted parameter fusion strategy that effectively preserves weak yet relevant knowledge embedded in the base model. Experimental results demonstrate that the proposed method significantly enhances merging performance in multi-task settings, thereby validating the critical role of the base model’s knowledge level in determining the success of model merging.

This work addresses the prevailing limitation in knowledge transfer between models of different sizes, where scaling up (S2L) and scaling down (L2S) are typically treated as incompatible tasks lacking a unified framework. To bridge this gap, we propose BoT, the first size-agnostic bidirectional scaling framework that treats model weights as continuous signals and leverages the discrete wavelet transform (DWT) and its inverse (IDWT) to enable parameter-free, computationally efficient knowledge transfer in both directions. Within this framework, S2L and L2S are naturally modeled as signal upsampling and downsampling, with the wavelet decomposition level serving as a dynamic scaling factor. Evaluated on DeiT, BERT, and GPT architectures, BoT achieves state-of-the-art performance on benchmarks such as GLUE and SQuAD while significantly reducing pretraining FLOPs—by up to 67.1% for S2L and 52.8% for L2S.

This work addresses the challenge of simultaneously preserving performance and adversarial robustness when compressing code models via single-source knowledge distillation. To overcome this limitation, the authors propose MoEKD, a novel framework that introduces the mixture-of-experts (MoE) mechanism into knowledge distillation for the first time. MoEKD employs a learnable routing strategy to dynamically aggregate knowledge from multiple expert teachers, enabling efficient and robust model compression. The approach breaks through the performance ceiling of conventional single-source distillation, achieving up to a 35.8% improvement in adversarial robustness and a 13% gain in prediction accuracy on vulnerability detection tasks compared to state-of-the-art methods, all while reducing model size by nearly half without compromising competitiveness.

This work addresses the challenges of high coordination overhead, fragmented context, and fragile sequential dependencies in multi-agent systems, as well as the lack of principled guidance in existing single-agent distillation approaches. The authors propose a two-stage adaptive distillation framework that introduces, for the first time, “Metric Freedom” (F) as a prior predictive indicator of skill utility. They demonstrate that F is governed by the topological rigidity of evaluation metrics and construct it by quantifying the coupling between output diversity and score variance via Mantel tests. Leveraging F, the framework dynamically selects between knowledge extraction and iterative optimization strategies. Experiments across four tasks, eleven datasets, and six metrics reveal a significant negative correlation between F and skill utility (ρ = −0.62, p < 0.05), with the proposed method achieving comparable or superior performance while reducing computational cost by up to 8× and latency by up to 15×.

This work addresses the challenge of automatically distilling structured, verifiable, and revisable skills from heterogeneous expert behavioral traces to construct personalized AI agents that embody human expertise, judgment, and interaction styles. The authors propose an end-to-end expert knowledge distillation framework that transforms multimodal source materials into versioned skill packages comprising capability trajectories—encompassing practices, mental models, and decision heuristics—and boundary trajectories capturing communication styles, interaction norms, and revision histories. This approach is the first to enable fully automated generation of structured AI skill packages that are portable, auditable, rollback-capable, and deployable across diverse agents, thereby overcoming key limitations of conventional prompt engineering and memory systems. An open-source implementation has garnered 18.5k GitHub stars, with 215 community-contributed skills from 165 contributors, demonstrating the feasibility and adoption of human-centric skill packaging.

This work proposes a modular expert recombination framework to address the limitations of existing model fusion approaches, which typically treat task-specific models as monolithic entities and lack component-level granularity and module reusability. The framework constructs a reusable library of component-level experts and employs a lightweight dynamic routing network to adaptively assemble an optimal sub-model at inference time based on the input. The fusion process is formulated as a bi-objective optimization problem, and a surrogate-assisted evolutionary algorithm efficiently searches for Pareto-optimal configurations. Extensive experiments demonstrate that the proposed method consistently outperforms strong baselines across diverse model scales, task types, and fine-tuning strategies, achieving superior generalization, inference efficiency, and storage economy.

The transition from monolithic language models to modular, skill-equipped agents marks a defining shift in how large language models (LLMs) are deployed in practice. Rather than encoding all procedural knowledge within model weights, agent skills -- composable packages of instructions, code, and resources that agents load on demand -- enable dynamic capability extension without retraining. It is formalized in a paradigm of progressive disclosure, portable skill definitions, and integration with the Model Context Protocol (MCP). This survey provides a comprehensive treatment of the agent skills landscape, as it has rapidly evolved during the last few months. We organize the field along four axes: (i) architectural foundations, examining the SKILL.md specification, progressive context loading, and the complementary roles of skills and MCP; (ii) skill acquisition, covering reinforcement learning with skill libraries (SAGE), autonomous skill discovery (SEAgent), and compositional skill synthesis; (iii) deployment at scale, including the computer-use agent (CUA) stack, GUI grounding advances, and benchmark progress on OSWorld and SWE-bench; and (iv) security, where recent empirical analyses reveal that 26.1\% of community-contributed skills contain vulnerabilities, motivating our proposed Skill Trust and Lifecycle Governance Framework -- a four-tier, gate-based permission model that maps skill provenance to graduated deployment capabilities. We identify seven open challenges -- from cross-platform skill portability to capability-based permission models -- and propose a research agenda for realizing trustworthy, self-improving skill ecosystems. Unlike prior surveys that broadly cover LLM agents or tool use, this work focuses specifically on the emerging skill abstraction layer and its implications for the next generation of agentic systems. Project repo: https://github.com/scienceaix/agentskills.