🤖 AI Summary
This study investigates the capacity of small language models (7B parameters or fewer) to effectively leverage external information in retrieval-augmented generation (RAG). Through systematic evaluation on models such as SmolLM2, Qwen2.5, and Llama 3.1—combined with BM25, E5-large-v2, and oracle retrievers across multiple prompt templates—the work introduces a novel parameterized knowledge partitioning framework that cleanly disentangles retrieval failure from context utilization failure for the first time. The findings reveal a fundamental bottleneck in small models’ ability to use retrieved content: even under oracle retrieval conditions, 85%–100% of samples fail to correctly incorporate the relevant answer, and 42%–100% of the model’s original knowledge is disrupted by the retrieved context. The dominant error mode is generation entirely unrelated to the provided context, indicating a pervasive inability to attend to or integrate external information.
📝 Abstract
Retrieval augmented generation RAG is widely deployed to improve factual accuracy in language models yet it remains unclear whether smaller models of size 7B parameters or less can effectively utilize retrieved information. To investigate this question we evaluate five model sizes from 360M to 8B across three architecture families SmolLM2 Qwen2.5 and Llama 3.1 under four retrieval conditions including no retrieval BM25 dense retrieval using E5 large v2 and oracle retrieval where the retrieved passage is guaranteed to contain the answer. We introduce a parametric knowledge split that separates questions a model can already answer from those that require external knowledge which allows us to isolate utilization failure from retrieval quality failure. We find three main results. First even with oracle retrieval models of size 7B or smaller fail to extract the correct answer 85 to 100 percent of the time on questions they cannot answer alone which indicates a fundamental utilization bottleneck. Second adding retrieval context destroys 42 to 100 percent of answers the model previously knew suggesting a distraction effect driven by the presence of context rather than its quality. Third an error analysis of 2588 oracle failures shows that the dominant failure mode is irrelevant generation where the model ignores the provided context entirely. These patterns hold across multiple prompt templates and retrieval methods. The results indicate that for models below 7B parameters the main limitation of RAG is context utilization rather than retrieval quality and that deploying RAG at this scale can lead to a net negative trade off under standard evaluation conditions.