🤖 AI Summary
This study addresses the trade-off in tabular in-context learning (ICL) between the accuracy degradation of fixed subsets and the low throughput of dynamic retrieval, proposing the QCOC method. QCOC introduces a novel commutativity-based operator compression mechanism that compiles contextual KV caches into shared compact prototypes. By integrating joint KV clustering with attention vector anchoring calibration and optimizing prototype values via closed-form solutions, it achieves efficient reuse while preserving accuracy. Experiments on OpenML datasets demonstrate that QCOC attains leading accuracy, yielding a 10.5× cache compression ratio and a 508× inference speedup over dynamic retrieval, effectively overcoming the accuracy-efficiency bottleneck.
📝 Abstract
Tabular in-context learning (ICL) has emerged as a training-free and accurate paradigm for tabular prediction, but current approaches to compressing its in-context examples face an accuracy-throughput tradeoff: fixed subsets can sacrifice accuracy, while query-specific retrieval limits cache reuse and batching across queries, reducing throughput. We propose QCOC (Query-Calibrated Operator Compression), which exploits the exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries. Instead of retaining raw examples, QCOC clusters their states into joint-KV prototypes, preserves per-cluster multiplicities and the original example count, and calibrates prototype values against attention query vectors produced by the in-context examples through an anchored closed-form solution. Prototype compression drives the speedup, while value fitting helps preserve accuracy. On 64 held-out OpenML-CC18 datasets, QCOC achieves the highest mean accuracy among the compared compression and retrieval methods at both retained counts. Across 12 configurations on seven long tables, it ranks first among compressed methods in ten and averages 0.23 percentage points below full context. Compressing 8,192 in-context examples to 512 memory slots yields a 10.5x cache compression ratio; excluding one-time compilation, in a single-core CPU online-serving comparison over 1,000 queries, QCOC is up to 508x faster than dynamic retrieval baselines and 1.98x faster than full-context inference. These results show that QCOC enables compact-memory reuse and efficient inference across queries while retaining accuracy close to full context.