TimeBill: Time-Budgeted Inference for Large Language Models

2025-12-25

AI Summary

For time-critical systems (e.g., robotics, autonomous driving), LLM inference must meet strict latency budgets while maintaining both response quality and task completion rate. This paper proposes the first KV cache adaptive management framework explicitly designed for real-time temporal constraints. Our approach comprises three key innovations: (1) a fine-grained Response Length Predictor (RLP) and Execution Time Estimator (ETE) that jointly model response length and latency at millisecond granularity; (2) dynamic KV cache eviction ratio adjustment guided by these predictions, enabling the first task- and time-aware adaptive caching; and (3) time-aware autoregressive decoding optimization. Experiments across diverse timeout policies demonstrate an average 23% reduction in timeout rate, significant improvement in task completion rate, and no degradation in response quality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are increasingly deployed in time-critical systems, such as robotics, autonomous driving, embodied intelligence, and industrial automation, where generating accurate responses within a given time budget is crucial for decision-making, control, or safety-critical tasks. However, the auto-regressive generation process of LLMs makes it challenging to model and estimate the end-to-end execution time. Furthermore, existing efficient inference methods based on a fixed key-value (KV) cache eviction ratio struggle to adapt to varying tasks with diverse time budgets, where an improper eviction ratio may lead to incomplete inference or a drop in response performance. In this paper, we propose TimeBill, a novel time-budgeted inference framework for LLMs that balances the inference efficiency and response performance. To be more specific, we propose a fine-grained response length predictor (RLP) and an execution time estimator (ETE) to accurately predict the end-to-end execution time of LLMs. Following this, we develop a time-budgeted efficient inference approach that adaptively adjusts the KV cache eviction ratio based on execution time prediction and the given time budget. Finally, through extensive experiments, we demonstrate the advantages of TimeBill in improving task completion rate and maintaining response performance under various overrun strategies.

Problem

• Addresses time-critical LLM deployment in robotics and autonomous systems.
• Solves adaptive KV cache eviction for varying time budgets.
• Balances inference efficiency with response performance under constraints.

Innovation

• Fine-grained response length and execution time prediction
• Adaptive KV cache eviction ratio adjustment based on time budget
• Balancing inference efficiency and response performance under constraints