Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstraction

2025-03-05

AI Summary

This work investigates whether large language models (LLMs) internally develop generalizable, linearly invariant conceptual abstractions—particularly concept vectors (CVs) required for analogical reasoning and their capacity to model higher-order abstract concepts (e.g., “previous/next”). Methodologically, it employs functional vector analysis, representational similarity analysis (RSA), attention head localization, and causal intervention. The study first identifies output-agnostic, cross-task-stable semantic CVs (e.g., antonym) and validates their interpretability and editability. However, it finds no evidence of linearly invariant representations for temporal abstraction concepts, revealing a structural limitation in LLMs’ higher-order abstract reasoning. This work establishes the first empirical framework for probing internal conceptual mechanisms in LLMs and provides critical boundary evidence on their representational capabilities.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsComputer Vision: Large Vision Models

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Analogical reasoning relies on conceptual abstractions, but it is unclear whether Large Language Models (LLMs) harbor such internal representations. We explore distilled representations from LLM activations and find that function vectors (FVs; Todd et al., 2024) - compact representations for in-context learning (ICL) tasks - are not invariant to simple input changes (e.g., open-ended vs. multiple-choice), suggesting they capture more than pure concepts. Using representational similarity analysis (RSA), we localize a small set of attention heads that encode invariant concept vectors (CVs) for verbal concepts like"antonym". These CVs function as feature detectors that operate independently of the final output - meaning that a model may form a correct internal representation yet still produce an incorrect output. Furthermore, CVs can be used to causally guide model behaviour. However, for more abstract concepts like"previous"and"next", we do not observe invariant linear representations, a finding we link to generalizability issues LLMs display within these domains.

Problem

• Explores whether LLMs have internal representations for analogical reasoning.
• Identifies invariant concept vectors for verbal concepts like 'antonym'.
• Highlights limitations in LLMs' ability to represent abstract concepts like 'previous' and 'next'.

Innovation

• Function vectors capture in-context learning tasks.
• Attention heads encode invariant concept vectors.
• Concept vectors causally guide model behavior.