ICA Lens: Interpreting Language Models Without Training Another Dictionary

2026-06-10

AI Summary

Current sparse autoencoders require training and storing large overcomplete dictionaries, which hinders efficient exploration of interpretable directions in language models. This work proposes ICALens, the first stable and efficient training-free ICA analysis pipeline that directly extracts non-Gaussian, human-interpretable directions from language model activations. The method integrates GPU-parallelized FastICA, LLM-specific stability strategies, diagnostic tools for model fit assessment, and a layer-wise analysis framework. Evaluated on GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, ICALens matches the performance of sparse autoencoders (SAEs) under low-budget sparse probing settings in SAEBench and demonstrates superior results in perturbation-based probing tasks.

Abstract

Finding interpretable directions in language-model representations is critical for understanding and controlling model behavior. Sparse autoencoders (SAEs) have become the standard tool for this purpose, but using them as the default first lens often requires training, storing, and evaluating large overcomplete dictionaries. This bottleneck limits rapid exploration and raises a fundamental question: how much interpretable structure is already visible from activation geometry before training another neural dictionary? Our intuition is simple: many interpretable directions are selective on tokens, and these directions should look less Gaussian than random directions. We therefore revisit independent component analysis (ICA), a classical method for finding non-Gaussian directions, as a compact lens for language-model interpretability. We find that ICA has been underestimated for LLM interpretability, because prior uses often relied on off-the-shelf ICA implementations that are brittle on LLM activations and lacked systematic tools for inspecting and evaluating the recovered directions. To bridge these gaps, we introduce ICALens, the first practical workflow for stable, efficient, and auditable ICA analysis of LLM representations. It combines an optimized GPU-parallel FastICA pipeline with LLM-specific stability recipes and better fitting diagnostics, enabling efficient and reliable layer-wise analysis. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, ICALens efficiently recovers compact, human-interpretable directions without per-layer gradient-based dictionary training. On SAEBench, ICA is competitive with public SAEs in sparse probing and outperforms them in targeted probe perturbation under small-to-medium budgets. These results suggest that ICA should not be viewed as a weak baseline, but as an efficient and complementary first lens for exploring language-model representations.

Problem

• interpretable directions
• language models
• activation geometry
• dictionary-free interpretability
• representation analysis

Innovation

• Independent Component Analysis
• Language Model Interpretability
• Sparse Autoencoders
• Activation Geometry
• FastICA