Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform

2025-07-31

AI Summary

This work addresses the high-dimensional redundancy inherent in word and sentence embeddings. We propose a semantic compression method based on Discrete Wavelet Transform (DWT), which captures hierarchical semantic structures via multi-scale wavelet decomposition—contrasting with conventional linear dimensionality reduction. To our knowledge, this is the first systematic application of DWT to embedding compression, offering both theoretical novelty and practical utility. Experiments demonstrate that embedding dimensions can be reduced by 50%–93% while preserving semantic similarity performance (degradation <0.5%). Moreover, downstream task accuracy improves by 1.2–2.8% on average across text classification, natural language inference (NLI), and semantic textual similarity (STS). The method is compatible with diverse pre-trained language models—including BERT, RoBERTa, and XLM-R—and exhibits strong cross-lingual generalization.

Technology Category

Data Mining & Knowledge Management: Data CompressionNatural Language Processing: Sentence-level Semantics, Textual Inference, etc.Machine Learning: Dimensionality Reduction/Feature Selection

Application Category

Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphsWeb Mining and Content Analysis: Robustness and generalizability of Web mining methodsSemantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful features from data. Tangible results from their application suggest that Wavelet transforms can be applied to NLP capturing a variety of linguistic and semantic properties. In this paper, we empirically leverage the application of Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We aim to showcase the capabilities of DWT in analyzing embedding representations at different levels of resolution and compressing them while maintaining their overall quality. We assess the effectiveness of DWT embeddings on semantic similarity tasks to show how DWT can be used to consolidate important semantic information in an embedding vector. We show the efficacy of the proposed paradigm using different embedding models, including large language models, on downstream tasks. Our results show that DWT can reduce the dimensionality of embeddings by 50-93% with almost no change in performance for semantic similarity tasks, while achieving superior accuracy in most downstream tasks. Our findings pave the way for applying DWT to improve NLP applications.

Problem

• Compress word and sentence embeddings using Discrete Wavelet Transform
• Maintain semantic quality while reducing embedding dimensionality
• Apply DWT to enhance NLP applications efficiently

Innovation

• Discrete Wavelet Transform compresses embeddings
• DWT maintains semantic similarity performance
• DWT reduces embedding dimensionality significantly