Field Aware Agent Skill Retrieval

2026-08-03

AI Summary

This work addresses the bottleneck faced by lifelong learning agents in retrieving relevant skills from large-scale skill repositories. To overcome this challenge, the authors propose a field-aware retrieval method that models each skill as a structured multi-field object—comprising fields such as name, description, and agent—and computes both sparse and dense similarity scores independently for each field. These field-specific similarities are then organized into a tensorized, field-aware representation and fused either through uniform weighting or a lightweight MLP, without concatenating the fields. This approach leverages only the structured representation to significantly enhance retrieval performance, achieving Recall@10 scores of 77.95 on SkillRet and 83.78 on SRA-Bench. Notably, the method’s advantage becomes more pronounced as the skill repository scales in size.

Abstract

As lifelong learning agents accumulate lifelong growing skill banks, retrieving the correct skill becomes an increasingly important bottleneck. Most current skill retrieval methods treat each skill as one flat document by concatenating fields such as the name, description, and body. However, skills are naturally structured, multi-field objects, where each field provides different information about when and how the skill should be used. In this work, we study whether preserving this structure improves skill retrieval. We represent each skill as its separate components, and compute sparse and dense similarities for each field independently, exposing a naturally tensorized, field-aware representation of the skill bank. We then combine these field-level scores either with uniform weights or with a small learned MLP. Across two different skill retrieval benchmarks, SkillRet and SRA-Bench, we find that keeping fields separate improves hybrid retrieval, and learning over the field-level scores gives the strongest and most consistent results. Our field-aware MLP reaches $77.95$ Recall@10 on SkillRet and $83.78$ Recall@10 on SRA-Bench, outperforming the corresponding concatenated learned baselines. We also find that the advantage grows as the skill bank becomes larger, suggesting that field-aware skill retrieval becomes especially useful in the setting where retrieval is most difficult. Our results show that skill representation itself matters, and that simply preserving the structure already present in skill files can substantially improve retrieval.

Problem

• skill retrieval
• structured representation
• lifelong learning
• field-aware retrieval
• hybrid retrieval

Innovation

• field-aware retrieval
• skill representation
• structured retrieval
• hybrid retrieval
• lifelong learning agents