Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)

2026-05-21

AI Summary

This work addresses the challenge of auditing implicit semantic subspaces and potentially harmful content within large language models, which are not directly inspectable through standard inference. The authors propose, for the first time, applying singular value decomposition (SVD) to the lm_head weight matrix, enabling static, inference-free model auditing with only five lines of PyTorch code. This approach reveals ethically problematic subspaces already embedded during pretraining. By leveraging vocabulary clustering, they introduce the Vocabulary Cluster Score (VCS) and Weighted Projection Score (WPS) to quantify subspace coherence and identify anomalous tokens. The method successfully reproduces known anomalies such as “shokubutsu-hyakka-tsu” and uncovers systematic differences in semantic structure and training data composition across models like GPT, Gemma, and Qwen. Notably, it demonstrates that alignment training fails to eliminate harmful content introduced during pretraining.

Abstract

We show that singular value decomposition of the lm_head} weight matrix of a transformer-based large language model -- requiring only five lines of PyTorch and no model inference -- reveals interpretable semantic subspaces directly from the model weights. Each left singular vector identifies the vocabulary tokens most readily selected when the hidden state aligns with the corresponding singular direction; inspecting these clusters exposes the model's training data composition and curation philosophy. Analysing GPT-OSS-120B, Gemma-2-2B, and Qwen2.5-1.5B, we find that singular value spectra and vocabulary cluster structures differ systematically across models: GPT exhibits a graduated hierarchy of functionally differentiated subspaces; Gemma is dominated by pre-nineteenth-century English orthography, forming a stepwise clustering structure that may contribute to high output controllability; and Qwen exhibits broad multilingual coverage alongside subspaces whose vocabulary the authors have determined to be ethically inappropriate for direct publication. Base-instruct comparison reveals that ethically concerning subspaces originate in pretraining and are not removed by post-training alignment. We introduce the Vocabulary Cluster Score (VCS) to quantify subspace coherence, and the Weighted Projection Score (WPS) as a static glitch token detector; applying WPS to GPT-OSS-120B recovers shokubutsu-hyakka-tsu (ID 137606), a well-known glitch token widely reported in the CJK language community, without any model inference. We propose a taxonomy of root causes for problematic vocabulary content and call for lm_head} SVD analysis to be adopted as a standard pre-release safety auditing step. Our findings further suggest directions toward SVD-guided tokenizer optimisation and more controllable LLM design.

Problem

• large language models
• ethically inappropriate content
• safety auditing
• vocabulary clusters
• pretraining artifacts

Innovation

• singular value decomposition
• lm_head analysis
• vocabulary clustering
• static safety auditing
• glitch token detection