Accurate and Efficient Statistical Testing for Word Semantic Breadth

2026-05-08

AI Summary

This study addresses the inflated Type I error rates in traditional dispersion-based semantic breadth hypothesis tests, which are susceptible to interference from differences in semantic directionality. To resolve this issue, the authors propose a Householder-aligned permutation test: first, Householder reflections are employed to align the mean directions of word embeddings, effectively neutralizing directional confounds; then, a non-parametric permutation test is conducted on the aligned contextualized word embedding clouds to accurately detect genuine differences in semantic breadth. This approach is the first to successfully disentangle semantic breadth from directional variation, achieving a 32.5% reduction in Type I error rates while maintaining high sensitivity to true semantic differences. Furthermore, GPU-accelerated batch processing yields a 23-fold speedup over CPU-based baselines.

Abstract

Measuring the breadth of a word's meaning, or its spread across contexts, has become feasible with contextualized token embeddings. A word type can be represented as a cloud of token vectors, with dispersion-based statistics serving as proxies for contextual diversity (Nagata and Tanaka-Ishii, ACL2025). These measurements are useful for deciding appropriate sense distinctions when constructing thesauri and domain-specific dictionaries. However, when comparing the breadth of two word types, naive hypothesis testing on dispersion can be misleading: differences in semantic direction can masquerade as dispersion differences, inflating Type-I error and yielding "statistically significant" outcomes even when there is no true breadth difference. This is problematic because significance testing should distinguish genuine effects from incidental fluctuations in small-difference regimes. We propose a Householder-aligned permutation test to isolate dispersion differences from directional differences. Our method applies a single Householder reflection to align the mean directions of the two word types and then performs a permutation test on the aligned token clouds, yielding calibrated, non-parametric p-values. For practicality, we introduce a GPU-oriented implementation that batches permutations and linear algebra operations. Empirically, our alignment reduced Type-I error by 32.5% while preserving sensitivity to genuine breadth differences, and achieved a 23x speedup over the CPU baseline.

Problem

• semantic breadth
• contextualized embeddings
• hypothesis testing
• Type-I error
• dispersion

Innovation

• semantic breadth
• Householder reflection
• permutation test
• contextualized embeddings
• Type-I error control