Efficient Iterative Retrieval with Heterogeneous Batching

2026-09-21

AI Summary

为解决信息检索系统中GPU利用率低的问题，提出Orthrus系统，通过异构批处理和统一推理循环优化计算模式，提高吞吐量并降低延迟。

Abstract

Modern information retrieval increasingly employs both embedding and generative models to handle complex queries. However, current serving systems suffer from low throughput and poor GPU utilization because they execute these models in isolation. Coarse-grained partitioning, such as dedicating GPUs to specific tasks, fails to adapt to dynamic workloads and creates computational "bubbles". To address these, we present Orthrus, a serving system that performs heterogeneous batching within a unified inference loop. The primary challenge lies in unifying embedding and generation workloads with conflicting computational patterns while optimizing batch composition for high performance. Orthrus addresses these challenges through chunked embedding with incremental pooling and by adjusting batch composition in a workload-aware manner. Evaluation on four A100 GPUs shows that, relative to baseline deployments, Orthrus achieves 1.28$\times$--4.52$\times$ higher throughput on controlled workloads and up to 55.8% lower end-to-end p99 latency on an iterative-RAG benchmark. We release our code at https://github.com/illinoisdata/Orthrus .

Problem

• information retrieval
• embedding models
• generative models
• throughput
• GPU utilization

Innovation

• heterogeneous batching
• unified inference loop
• chunked embedding with incremental pooling