Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder

2025-06-24

AI Summary

This work addresses the gap between symbolic and distributional semantics to enhance interpretability, controllability, compositional generalization, and robustness in autoregressive Transformer language models. Method: We propose a unified learning framework integrating compositional semantics with distributed representations, built upon variational autoencoders (VAEs), vector-quantized VAEs (VQ-VAEs), and sparse autoencoders (SAEs). Systematically investigating how latent-space geometry affects semantic compositionality and interpretability, we model structured geometric properties of latent variables from a compositional-semantic perspective, uncovering mappings between latent-space topology and linguistic meaning. Contribution/Results: Experiments demonstrate that structured latent representations significantly improve semantic disentanglement and controllable generation. Our approach provides both theoretical foundations and architectural blueprints for next-generation semantic models that reconcile symbolic rigor with distributional robustness.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Integrating compositional and symbolic properties into current distributional semantic spaces can enhance the interpretability, controllability, compositionality, and generalisation capabilities of Transformer-based auto-regressive language models (LMs). In this survey, we offer a novel perspective on latent space geometry through the lens of compositional semantics, a direction we refer to as extit{semantic representation learning}. This direction enables a bridge between symbolic and distributional semantics, helping to mitigate the gap between them. We review and compare three mainstream autoencoder architectures-Variational AutoEncoder (VAE), Vector Quantised VAE (VQVAE), and Sparse AutoEncoder (SAE)-and examine the distinctive latent geometries they induce in relation to semantic structure and interpretability.

Problem

• Bridging compositional and distributional semantics in language models
• Exploring latent space geometry via autoencoder architectures
• Enhancing interpretability and controllability of semantic representations

Innovation

• Integrates compositional and symbolic semantics
• Surveys VAE, VQVAE, SAE architectures
• Bridges symbolic and distributional semantics