Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance

2026-10-08

AI Summary

This study addresses the limitation that existing evaluations of large language model (LLM) reasoning rely solely on behavioral performance, obscuring underlying organizational mechanisms. Inspired by neuroscience, we construct functional graphs based on attention head activation similarities and establish small-world connectivity as a structural signature of LLM reasoning for the first time, revealing a significant positive correlation between the small-world index and fluid reasoning performance. Furthermore, by integrating core and bridge centrality scores to identify critical attention heads, we propose SWA, a hierarchical sparse allocation strategy grounded in community connectivity features. Experiments across six LLMs demonstrate that SWA outperforms competitive baselines, reducing WikiText perplexity by up to 20% while effectively balancing model performance with the preservation of small-world topological structure.

Abstract

Understanding large language model (LLM) reasoning requires looking beyond behavioral performance to examine how reasoning ability is reflected in internal organization. Inspired by neuroscience findings linking higher intelligence to stronger small-world organization in functional brain networks, we investigate small-world connectivity as a structural signature of LLM reasoning. We construct functional graphs from attention-head activation similarities and find that a higher small-world index (SWI), capturing local clustering and short global paths, consistently correlates with better fluid reasoning performance across models and training checkpoints. Since local clustering is central to small-world organization, we further examine how heads important for model performance connect within and across communities. We find that these heads tend to have a larger share of connection weight within their own communities (high core scores) and a more concentrated weight distribution across communities (low bridge scores). These observations motivate the hypothesis that high core and low bridge scores serve as structural indicators of head importance for reasoning capability. We validate this hypothesis through pruning, introducing Small-World Allocation (SWA), a hierarchical sparsity allocation method guided by these scores. Across six LLMs, SWA better preserves small-world organization and model performance than competing allocation strategies, reducing WikiText perplexity by up to 20%. Together, these findings identify small-world functional connectivity as a measurable signature of LLM reasoning performance, offering a structural perspective that complements behavioral evaluation.

Problem

• Large Language Models
• Reasoning Performance
• Small-World Connectivity
• Internal Organization
• Functional Graphs

Innovation

• Small-World Connectivity
• Functional Graph
• Attention Head Pruning
• Hierarchical Sparsity Allocation
• Reasoning Performance