token accounting

This study addresses the limitations of large language model (LLM)-based multi-agent systems in software engineering, particularly the lack of transparency in resource consumption, unpredictability of costs, and unclear environmental impact. To this end, it introduces the first standardized token consumption evaluation framework tailored for agent-based software engineering. By analyzing execution trajectories from the ChatDev framework across 30 development tasks, the work maps internal agent interactions to standard software engineering phases—design, coding, completion, code review, testing, and documentation—and quantifies the distribution of input, output, and reasoning tokens across these stages. The analysis reveals that the code review phase alone accounts for 59.4% of total token usage, with input tokens comprising 53.9% of the total, indicating that cost is primarily driven by automated refinement and validation rather than initial code generation. These findings provide empirical foundations for optimizing workflows, forecasting costs, and designing efficient agent collaboration protocols.

Prior evaluations of large language model (LLM) prompting strategies have predominantly focused on task performance, neglecting computational efficiency—specifically, the trade-off between performance and token consumption. Method: This paper introduces an efficiency-oriented evaluation paradigm, proposing the first Big-Oₜₒₖ theoretical framework based on progressive complexity to characterize the asymptotic token growth order of prompting strategies; it further designs the empirical metric *Token Cost* to quantify diminishing marginal returns in performance as token count increases. Combining theoretical modeling with systematic empirical measurement, the study evaluates mainstream strategies—including zero-shot, few-shot, and chain-of-thought prompting. Results: Experiments reveal substantial token redundancy across most prompting strategies; notably, optimal strategies achieve ≥90% of peak task performance while reducing token consumption by 30–60%. This work establishes a quantifiable, comparable benchmark for efficient prompt engineering.

While long contexts enhance the performance of large language models, they incur substantial computational and economic costs, and existing compression methods lack a unified, deployment-oriented evaluation framework. This work formulates context strategy selection as a deployment-aware optimization problem that jointly considers task performance, token cost, and preprocessing reuse. It introduces the first efficiency frontier framework integrating these three dimensions, enabling systematic comparison and dynamic decision-making under multidimensional constraints. Leveraging an amortized cost model, the proposed joint optimization method combines retrieval and memory compression. Empirical results on HotpotQA demonstrate a 25% reduction in effective token usage while maintaining an F1 score of approximately 0.78; in high-performance regimes, amortized memory compression reduces token costs by over 50% compared to full-context prompting.

This study investigates whether increasing the number of training tokens yields commensurate improvements in model performance and, for the first time, incorporates energy consumption and training duration into the evaluation framework. Under fixed hardware conditions, the authors conduct repeated experiments by training the 1.1B-parameter TinyLlama model on identical GPUs, combining high-frequency power sampling with repeated-measures ANOVA to systematically assess the joint impact of token scale on performance, energy use, and training time. The findings reveal diminishing returns in conventional performance metrics with increasing token counts, while energy–time efficiency exhibits a strictly monotonic decline, indicating that high-token training can substantially reduce energy efficiency. This work introduces an energy-aware perspective on parameter efficiency, offering empirical foundations for greener large language model training.

This work addresses the frequent neglect of efficiency costs—such as latency, token consumption, and execution steps—in the deployment of large language model (LLM) agents. It formulates agent efficiency as a Pareto frontier problem balancing effectiveness and cost, and introduces key techniques centered on three core components: memory, tool learning, and planning. Specifically, the paper proposes context compression and management, reinforcement learning–based minimization of tool calls, and controlled search strategies. The authors establish the first unified efficiency-oriented evaluation benchmark, integrating multidimensional efficiency metrics and defining component-specific evaluation protocols. Their approach significantly reduces resource overhead while maintaining task performance, yielding generalizable design principles for efficient LLM agents across diverse methods.

This work addresses a critical flaw in existing LLM inference benchmarks, where single-process clients under high concurrency suffer from Python’s Global Interpreter Lock (GIL), leading to severe distortion in Time-to-First-Token (TTFT) and Time Per Output Token (TPOT) metrics. The study is the first to model the client as an M/G/1 queue, uncovering systematic bias introduced by queuing effects. To rectify this, the authors propose a multi-process, unbiased evaluation framework accompanied by a normalized metric—Normalized Time Per Output Token (NTPOT). This approach effectively eliminates client-side bottlenecks, enabling accurate and reproducible performance evaluation at scale, supporting thousands of queries per second. It substantially reduces the latency overestimation—often several-fold—in conventional benchmarks, thereby reflecting the true performance of production-grade LLM services.

This work addresses the challenges of context overflow, outdated state tracking, and escalating reasoning costs in enterprise workflows caused by verbose tool responses from large language model (LLM) agents. Focusing on a Microsoft Dynamics 365 expense reimbursement scenario, the authors propose a context management approach that integrates recent tool interaction pruning with automated summarization. Leveraging GPT-5 and Claude Sonnet 4.5 models via the Model Context Protocol, the method enables efficient context compression and multidimensional evaluation. Experimental results demonstrate that the approach achieves a 91.6% task completion rate and 99.64% monetary coverage using only 550k tokens and 5.79 hours of compute, significantly outperforming a full-history baseline while substantially reducing resource consumption without compromising performance.

This work addresses the common oversight in existing large language model (LLM) distillation methods of neglecting end-to-end energy consumption on the teacher model side—encompassing data generation, logit caching, and evaluation. The study proposes the first systematic framework for accounting energy costs, employing fine-grained GPU power monitoring to empirically measure energy use and carbon emissions across the entire distillation pipeline. It quantifies previously hidden energy costs per stage, constructs an energy–quality Pareto frontier, and introduces distillation strategies and hyperparameter selection criteria tailored to energy-efficiency constraints. To support reproducible green AI research, the authors open-source a standardized measurement protocol and toolchain, revealing high-cost components overlooked by conventional approaches and establishing a benchmark for sustainable LLM development.

This work addresses the high context token consumption in software engineering agents caused by processing lengthy source code, which hinders deployment efficiency and increases costs. Building upon and extending the state-in-context framework, we introduce, for the first time, a systematic approach to semantic-preserving source code minification that reduces input length by removing or compressing non-essential syntactic elements. We integrate multiple minification strategies into the DirectSolve agent and evaluate them end-to-end using GPT-5-mini and GPT-4.1. Experimental results demonstrate that our method reduces input tokens by 42% on average while incurring only a 12-percentage-point drop in task success rate, confirming that lightweight input transformation can substantially improve cost-efficiency without severely compromising performance.