SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge

2026-07-29

AI Summary

This work addresses the modality gap in existing agent systems, which struggle to effectively integrate textual knowledge with parametric skills, thereby limiting task performance. To bridge this divide, the paper proposes treating model weights as a novel modality natively amenable to reasoning by large language models (LLMs), unifying parametric skills and textual knowledge into a composable representation for the first time. The approach employs prefix tuning to construct parametric skills, designs an LLM-augmented architecture, and introduces an instruction-driven weight composition mechanism to enable cross-modal skill generation and transfer. Experimental results demonstrate that the method significantly outperforms baselines relying solely on text or weights, achieving performance gains in multitask settings that are unattainable by single-modality approaches.

Abstract

Agentic systems driven by large language models (LLMs) regularly feature two key mechanisms to autonomously solve complex problems: synthesizing text-based knowledge and procedures from past experiences and building parametric (weight-space) skill libraries for recurring sub-goals. To date, research has largely treated these as orthogonal pursuits: either organizing textual knowledge through composition and reflection, or consolidating parametric skills via weight-space merging. Consequently, the seamless integration of text and model weights for targeted performance improvements remains largely unexplored. This work bridges this modality gap by treating model weights as an additional modality that an LLM can natively reason over. We instantiate parametric learning via prefix-tuning and augment an LLM to ingest both prefix weights and rich textual data which capture relationships to a target capability. Our augmented LLM, which we call SkillSmith, synthesizes these inputs to perform instruction-steered parametric synthesis, directly outputting new prefix weights that manifest the target skill. We demonstrate that our approach significantly outperforms both text-only and weight-space-only baselines, unlocking performance gains that are out of reach for uni-modal (text-only or weight-only) adaptations.

Problem

• parametric skills
• textual knowledge
• modality integration
• LLM reasoning
• skill composition

Innovation

• parametric skill synthesis
• prefix-tuning
• multimodal reasoning
• instruction-steered learning
• weight-space composition