context compression

Large language models (LLMs) face significant challenges in data-intensive tasks—such as database querying and developer observability—including excessive input length, high noise, and prohibitive token consumption. Method: This paper formalizes the token budget as a finite attention resource and introduces *task-aware input text reduction* as a first-class design principle in language-data systems. We propose an adaptive text reduction pipeline that jointly leverages task semantics and token-budget awareness during preprocessing, enabling goal-directed information preservation—not generic compression. Contribution/Results: Experiments demonstrate substantial reductions in computational overhead and carbon footprint, alongside improved query accuracy and system scalability. Our approach establishes a new paradigm for sustainable, high-performance LLM–database co-processing in noisy, large-scale data environments.

To address memory bandwidth and capacity bottlenecks limiting inference efficiency in large language models (LLMs), this work proposes an LLM-aware on-chip memory controller architecture. The method integrates hardware-accelerated lossless block compression (LZ4/ZSTD) with fine-grained bit-level accessibility control in the memory controller, synergistically coordinated with a context-adaptive dynamic quantization engine for weights and KV caches. This enables joint compression–quantization optimization, preserving inference accuracy without loss while dynamically scaling memory bandwidth and energy consumption according to input context length. Experimental results demonstrate 25.2% reduction in weight storage and 46.9% reduction in KV cache size. Implemented in a 7 nm process, the prototype achieves 8 TB/s throughput at 4 GHz with 32 lanes, incurring an area overhead of less than 3.8 mm².

This work addresses the lack of explicit preservation and verifiability of semantic commitments—such as goals, constraints, and decisions—in existing context compression methods for large language models. The authors propose the Context Codec framework, which models dialogue states as typed, source-anchored semantic atoms and decomposes compression into five modular stages: extraction, normalization, representation, rendering, and verification. To evaluate fidelity, they introduce novel metrics including semantic compression error taxonomy and critical atom recall, alongside a conservative fallback strategy. They also design the Context Compression Language (CCL), a compact, human-readable, and auditable format, with CCL-Core striking a balance between structured text and JSON to achieve superior performance in terms of size, risk mitigation, and verifiability.

Existing LLM compression evaluations focus solely on language modeling and understanding tasks, overlooking the impact of compression on agent capabilities—such as workflow generation, tool invocation, long-context reasoning, and real-world application performance. This paper introduces ACBench, the first benchmark explicitly designed to assess LLM compression effects on agent-level competencies, systematically evaluating four core agent task categories. We innovatively formalize and quantify the *non-uniformity* of compression effects, proposing three novel analytical metrics: ERank, Top-k Ranking Correlation, and Energy. Empirical evaluation spans 15 mainstream models (Gemma-2B to Qwen2.5-32B and DeepSeek-R1-Distill), applying GPTQ/AWQ quantization and Wanda/SparseGPT pruning. Results show that 4-bit quantization degrades workflow and tool-use performance by only 1–3%, yet reduces real-world accuracy by 10–15%. ACBench is publicly released to enable reproducible, evidence-based compression strategy selection.

To address the degradation of semantic fidelity in large language models (LLMs) under extreme compression, this paper proposes a semantics-preserving compression framework that synergistically integrates pruning and quantization. Methodologically, it combines structured pruning, INT4/INT2 quantization, semantic-sensitive layer adaptation, and knowledge distillation–based fine-tuning. A key contribution is the novel metric “Semantic-preserving Compression Ratio” (SrCr), which formally models the trade-off between compression ratio and semantic fidelity—enabling principled, joint configuration optimization. Experiments demonstrate that, at equal theoretical compression ratios, the proposed joint scheme improves average downstream task performance by 20% over pure quantization baselines, while maintaining over 98% original semantic consistency.

This work addresses the inefficiency and low utilization of large language model (LLM) inference under the “memory wall,” particularly in scenarios involving long outputs, low computational intensity, and stringent latency constraints. To overcome these limitations, the authors propose RPU, a chiplet-based architecture optimized for inference, which innovatively integrates capacity-optimized HBM (HBM-CO), a bandwidth-first scalable chiplet design, and a microarchitecture that decouples computation, memory, and communication. This co-design significantly enhances memory bandwidth utilization. Evaluated on the Llama3-405B model, RPU achieves up to a 45.3× reduction in latency and an 18.6× improvement in throughput compared to an H100 system under the same thermal design power (TDP) envelope.

This work addresses the challenge of simultaneously achieving low latency and high throughput in highly dynamic networks, where traditional congestion control algorithms often fall short. It presents the first systematic exploration of leveraging large language models (LLMs) for congestion control, proposing an LLM-driven strategy that integrates a dynamic triggering mechanism with carefully constrained control freedom. The approach initially focuses on the congestion avoidance phase and is subsequently extended to multi-phase coordinated control, enabled through tailored instruction engineering and deployed via network simulation. Experimental results demonstrate that, across both static and dynamic network traces, the proposed method reduces latency by up to 50% compared to conventional algorithms while incurring less than 0.3% throughput loss, thereby significantly enhancing overall performance in dynamic environments.

Edge devices are constrained by limited memory and computational resources, making it challenging to deploy large language models. To address this, this work proposes MeKi, a novel system that introduces a model scaling paradigm decoupling model capacity from computational cost by substituting FLOPs growth with read-only memory (ROM) storage. MeKi embeds memory-based expert modules within each Transformer layer and employs reparameterization to compress trainable parameters into static lookup tables, thereby injecting pre-stored semantic knowledge without incurring any inference latency overhead. Experimental results demonstrate that MeKi significantly enhances generation performance while maintaining inference speed comparable to that of dense models, validating the effectiveness and practicality of memory-based model expansion.

This study addresses the challenge of efficiently compressing text generated by large language models (LLMs), achieving substantial improvements in compression ratios under both lossless and lossy settings. The authors propose a novel question-answering–based interactive compression protocol inspired by the “twenty questions” game, which narrows the performance gap between small and large models using only ten binary queries. By integrating domain-adapted LoRA fine-tuning, prompt rewriting, and arithmetic coding, the framework enables end-to-end optimization. Experimental results demonstrate a two-fold improvement in lossless compression efficiency, a lossy compression ratio of 0.03, and an exceptionally low interactive compression ratio ranging from 0.0006 to 0.004—surpassing existing methods by over two orders of magnitude.

This work addresses the inherent trade-off between latency and throughput in deploying dense large language models (e.g., Llama-3.1-70B/405B), particularly when model size exceeds device memory, where parallelization strategies critically impact performance. The study systematically evaluates tensor parallelism (TP), pipeline parallelism (PP), and their hybrid configurations for single-node inference, revealing that TP is more effective at reducing latency while PP better enhances throughput. Building on this insight, the authors propose dynamically adjusting the TP–PP ratio to finely tune the latency–throughput trade-off. Experimental results demonstrate that this approach enables system optimization tailored to diverse service objectives—such as meeting strict SLA requirements or maximizing throughput—thereby offering clear architectural guidance for deploying dense LLMs in practice.