MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference

2026-07-03

AI Summary

为降低大型语言模型推理成本，提出CacheSpec框架，利用小型模型进行语义变量提取和推测草稿生成，提高缓存复用率和处理速度。

Abstract

Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost. We present MiniCache, a reusable program caching framework that transforms Program-of-Thought (PoT) programs into parameterized cache objects, enabling reusable computation across structurally similar requests. MiniCache reuses the same small model for semantic variable extraction on cache-hit requests and speculative drafting during target-LLM generation, reducing expensive target-LLM invocations while preserving task quality. Experiments on shopping-style request datasets, WebShop, Formula, and CodeTAT-QA demonstrate that MiniCache improves the trade-off between inference latency, cache reuse, and accuracy, achieving up to 3.1x lower latency and 2.8x higher throughput under parallel serving. These results show that small models are most effective not as replacements for large models, but as lightweight interface models that enable reliable and efficient reusable program caching.

Problem

• large language models
• inference cost
• program-level caching
• small models
• reusable computation logic

Innovation

• program-level caching
• small models
• inference optimization
• reusable computation logic
• semantic variable extraction