Lightweight Chunk Selection for Mobile Retrieval-Augmented Generation

2026-08-04

AI Summary

This work addresses the trade-off between high computational overhead and insufficient evidence when deploying Retrieval-Augmented Generation (RAG) systems on mobile devices. It formulates lightweight text chunk selection as an evidence alignment problem, uniquely integrating large language model (LLM) query intent, Mixture-of-Experts (MoE) routing signals, and retrieval embeddings. A compact multilayer perceptron constructs evidence prototypes in embedding space, and the optimal chunk is selected based on cosine similarity. The approach innovatively introduces semantic correctness labels grounded in evidence sufficiency and a task-aware feature selection strategy. Experimental results demonstrate that, under stringent resource constraints, the method improves rank-1 evidence selection accuracy by 2.5% on average while maintaining strong parameter efficiency and practical applicability.

Abstract

RAG improves the factual grounding of LLM by incorporating external knowledge, but deploying RAG on mobile and edge devices remains challenging because retrieved context increases computation and memory. A direct way to reduce this cost is to retain only one retrieved chunk before generation, but the top-ranked retrieved chunk is not always the most evidence-supporting one, since retrieval similarity does not necessarily imply evidential sufficiency. Existing context-reduction methods can improve context quality, but often require additional LLMs or compressors that are costly under a strict mobile budget. In this paper, we study lightweight RAG chunk selection as an evidence-alignment problem. Our selector combines three complementary feature sources: question hidden states that represent LLM-side query intent, MoE routing-derived expert signals that capture the generator's internal routing structure, and retrieved chunk embeddings that preserve candidate-side evidence geometry. A compact multilayer perceptron maps these features to an evidence prototype in the chunk embedding space, and the candidate most aligned with this prototype is selected by cosine similarity. For stricter deployment budgets, we further introduce an optional task-aware feature selection strategy to reduce the selector input dimension. To support supervised evaluation, we construct semantic chunk-correctness labels based on evidence sufficiency rather than answer-string containment. Experiments show that the proposed selector consistently improves rank-1 evidence selection over mobile-applicable baselines by an average of 2.5%. These results suggest that using LLM-side query representations and MoE routing information and aligning them with retrieval-side candidate embedding is an effective and parameter-efficient strategy for mobile-applicable RAG chunk selection.

Problem

• Retrieval-Augmented Generation
• Mobile RAG
• Chunk Selection
• Evidence Alignment
• Lightweight Deployment

Innovation

• lightweight RAG
• chunk selection
• evidence alignment
• MoE routing
• mobile deployment