CompactPrompt: A Unified Pipeline for Prompt Data Compression in LLM Workflows

2025-10-20

AI Summary

To address the high inference cost of large language models (LLMs) in agent workflows caused by lengthy prompts and multi-source data streams, this paper proposes an end-to-end prompt-and-data co-compression framework. Methodologically, it innovatively integrates hard prompt compression—pruning low-information tokens via self-information scoring and dependency-aware phrase grouping—with lightweight file-level compression—applying n-gram abbreviation for textual data and uniform quantization for numerical data—to jointly handle heterogeneous text and numeric inputs. The framework further supports real-time visualization of compression decisions and cost–performance Pareto analysis. Evaluated on benchmarks including TAT-QA and FinQA, it achieves up to 60% reduction in token usage and inference cost, while maintaining output quality degradation of less than 5% for Claude-3.5-Sonnet and GPT-4.1-Mini—significantly outperforming existing baselines.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) deliver powerful reasoning and generation capabilities but incur substantial run-time costs when operating in agentic workflows that chain together lengthy prompts and process rich data streams. We introduce CompactPrompt, an end-to-end pipeline that merges hard prompt compression with lightweight file-level data compression. CompactPrompt first prunes low-information tokens from prompts using self-information scoring and dependency-based phrase grouping. In parallel, it applies n-gram abbreviation to recurrent textual patterns in attached documents and uniform quantization to numerical columns, yielding compact yet semantically faithful representations. Integrated into standard LLM agents, CompactPrompt reduces total token usage and inference cost by up to 60% on benchmark dataset like TAT-QA and FinQA, while preserving output quality (Results in less than 5% accuracy drop for Claude-3.5-Sonnet, and GPT-4.1-Mini) CompactPrompt helps visualize real-time compression decisions and quantify cost-performance trade-offs, laying the groundwork for leaner generative AI pipelines.

Problem

• Reduces LLM token usage and inference costs
• Compresses prompts while preserving output quality
• Optimizes agent workflows with data compression techniques

Innovation

• Compresses prompts by pruning low-information tokens
• Applies n-gram abbreviation to recurrent textual patterns
• Uses uniform quantization for numerical data compression