🤖 AI Summary
To address poor generalization in high-dimensional tabular data classification—caused by excessively high embedding dimensions and scarce labeled samples—this paper proposes Partitioned-LDA, a linear discriminant dimensionality reduction method integrating blockwise covariance estimation with shrinkage regularization. It partitions high-dimensional word embeddings into non-overlapping blocks, estimates local covariance matrices per block, and applies shrinkage correction to mitigate estimation bias under small-sample conditions, thereby enhancing the stability and discriminative power of Linear Discriminant Analysis (LDA). Experiments demonstrate that even a 2-dimensional Partitioned-LDA projection surpasses the classification accuracy of the original high-dimensional embeddings on public benchmark leaderboards, consistently ranking among the top ten. Compared to standard PCA and shrinkage-regularized LDA, Partitioned-LDA exhibits superior robustness and higher dimensionality-reduction efficiency under limited training data. This work establishes a novel, interpretable, lightweight, and high-performance embedding compression paradigm for low-resource tabular classification.
📝 Abstract
The Engineers' Salary Prediction Challenge requires classifying salary categories into three classes based on tabular data. The job description is represented as a 300-dimensional word embedding incorporated into the tabular features, drastically increasing dimensionality. Additionally, the limited number of training samples makes classification challenging. Linear dimensionality reduction of word embeddings for tabular data classification remains underexplored. This paper studies Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). We show that PCA, with an appropriate subspace dimension, can outperform raw embeddings. LDA without regularization performs poorly due to covariance estimation errors, but applying shrinkage improves performance significantly, even with only two dimensions. We propose Partitioned-LDA, which splits embeddings into equal-sized blocks and performs LDA separately on each, thereby reducing the size of the covariance matrices. Partitioned-LDA outperforms regular LDA and, combined with shrinkage, achieves top-10 accuracy on the competition public leaderboard. This method effectively enhances word embedding performance in tabular data classification with limited training samples.