SkillSight: Seeing Through Shared Descriptions for Accurate Skill Retrieval

2026-07-21

AI Summary

Current large language model agents often suffer from limited retrieval performance due to overlooking the shared contextual structure in skill descriptions, which obscures task-relevant signals. This work proposes SkillSight, a training-free retrieval framework that systematically identifies and calibrates such shared contextual bias for the first time. SkillSight constructs a semantic background subspace by identifying generic terms via inverse document frequency (IDF) and applies lexical-level evidence weighting to doubly recalibrate dense retrieval results. Experiments demonstrate that SkillSight improves Recall@10 by up to 20.21 percentage points on SRA-Bench and SkillBench-Supp, surpasses LLM Selection by 4.97 percentage points in end-to-end performance, and achieves a 1,248× speedup over Dense+Reranker during inference.

Abstract

As large language model agents gain access to increasingly large skill libraries, retrieving the right skill becomes critical to reliable capability selection and execution. Existing retrievers often treat skill descriptions as ordinary documents, overlooking their highly regular structure: shared descriptive patterns recur across many skills while providing little evidence for distinguishing the required capability. We show that this shared descriptive background systematically contributes to dense relevance scores, induces a pronounced energy gap between queries and skill documents, and obscures task-relevant signals. Based on this observation, we propose SkillSight, a training-free retrieval framework that calibrates shared background in both semantic and lexical spaces. Semantic Background Calibration estimates a background subspace from generic tokens identified by IDF, reducing similarity induced by shared descriptive patterns, while Lexical Evidence Calibration downweights shared background tokens to recover discriminative token-level evidence. Experiments on SRA-Bench and SkillBench-Supp demonstrate consistent improvements across retrieval metrics, with SkillSight improving Recall@10 by up to 20.21 percentage points over the original dense retriever. In end-to-end evaluation, SkillSight achieves the best overall performance across three agent models and outperforms LLM Selection by up to 4.97 percentage points. It is also up to 1,248 times faster than the Dense + Reranker baseline. These results identify shared descriptive background as a key source of bias in skill retrieval and demonstrate that explicitly calibrating it enables accurate and efficient skill selection without additional training. Our code is available at https://github.com/xiaojinying/SkillSight.

Problem

• skill retrieval
• shared descriptive background
• relevance scoring
• capability selection
• dense retrieval

Innovation

• shared descriptive background
• semantic background calibration
• lexical evidence calibration
• training-free retrieval
• skill retrieval