ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations

2026-04-08

AI Summary

This work addresses the lack of systematic tools for uncovering the relationship between representations learned by neural networks—particularly tabular foundation models—and human-interpretable concepts. To bridge this gap, we introduce ConceptTracer, the first interactive analysis framework that integrates information-theoretic measures of concept saliency and selectivity, specifically designed for tabular foundation models such as TabPFN. By quantifying both the strength and specificity of individual neuron responses to predefined concepts and coupling these metrics with an intuitive visual interface, ConceptTracer enables users to efficiently identify neurons with clear semantic interpretations. Experiments on TabPFN successfully reveal neurons whose activations align closely with human-understandable concepts, demonstrating the effectiveness and practical utility of our approach in elucidating how neural networks encode conceptual knowledge.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neural networks deliver impressive predictive performance across a variety of tasks, but they are often opaque in their decision-making processes. Despite a growing interest in mechanistic interpretability, tools for systematically exploring the representations learned by neural networks in general, and tabular foundation models in particular, remain limited. In this work, we introduce ConceptTracer, an interactive application for analyzing neural representations through the lens of human-interpretable concepts. ConceptTracer integrates two information-theoretic measures that quantify concept saliency and selectivity, enabling researchers and practitioners to identify neurons that respond strongly to individual concepts. We demonstrate the utility of ConceptTracer on representations learned by TabPFN and show that our approach facilitates the discovery of interpretable neurons. Together, these capabilities provide a practical framework for investigating how neural networks like TabPFN encode concept-level information. ConceptTracer is available at https://github.com/ml-lab-htw/concept-tracer.

Problem

• neural representations
• concept saliency
• concept selectivity
• tabular foundation models
• interpretability

Innovation

• ConceptTracer
• concept saliency
• concept selectivity
• neural representations
• tabular foundation models