LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents

2026-06-04

AI Summary

This work addresses the high contextual overhead and explicit exposure of skill descriptions when large language model agents invoke textual skills. The authors propose a novel pre-trained hypernetwork-based approach that, for the first time, maps textual skills into weight space to generate plug-and-play LoRA adapters, enabling efficient skill invocation without requiring the skill description as input at each step. This method supports modular loading, scaling, and composition of skills, substantially reducing exposure risk while inducing a structured semantic geometry in parameter space. Experiments demonstrate significant improvements: success rates increase by 21.4 and 13.4 percentage points on ALFWorld and Search-QA, respectively, with a 3.0-point gain in Exact Match accuracy, alongside a 64.1%–72.2% reduction in skill-related token consumption.

Abstract

Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext. We present LatentSkill, a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. LatentSkill stores skill knowledge in weight space rather than context space, removing per-step skill tokens while preserving modular loading, scaling, and composition. On ALFWorld and Search-QA, LatentSkill outperforms the corresponding in-context skill baseline while using substantially fewer prefill tokens: it improves ALFWorld success by 21.4 and 13.4 points on the seen and unseen splits with 64.1% fewer prefill tokens, and improves Search-QA exact match by 3.0 points with 72.2% lower skill-token overhead. Further analysis shows that generated skill LoRAs form a structured semantic geometry, can be precisely controlled via the LoRA scaling coefficient, and can be composed through parameter-space arithmetic when skill components are aligned. These findings suggest that weight-space skills provide an efficient, modular, and less exposed substrate for extending LLM agents.

Problem

• LLM agents
• textual skills
• context overhead
• skill injection
• plaintext exposure

Innovation

• LatentSkill
• LoRA
• weight-space skills
• modular composition
• hypernetwork