When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks

2025-02-04

AI Summary

This work investigates the impact of text embedding compression on large language model (LLM) regression performance under heterogeneous noise levels and causal strength—specifically in financial return prediction, writing quality assessment, and review rating. We propose a lightly supervised autoencoder-based embedding compression method and conduct systematic cross-domain evaluation. Our results reveal a “dual regularization effect”: moderate compression significantly mitigates overfitting—reducing average MAE by 12.7% in high-noise tasks like financial forecasting—but harms performance in strongly causally structured tasks. We further demonstrate that compression efficacy critically depends on task signal-to-noise ratio and causal dependency structure, and we interpret the mechanism via interpretable representation dimensions (e.g., sentiment). This study establishes theoretical boundaries for embedding compression applicability and provides principled guidance for enhancing LLM representation robustness.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model's text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder's hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.

Problem

• Explores embedding compression in LLM regression tasks.
• Compares performance across financial, writing, and review contexts.
• Assesses compression's impact on overfitting and task performance.

Innovation

• LLM embedding compression
• autoencoder hidden representation
• minimally supervised manner