neural head architecture

When large language models serve as knowledge bases, their internal mechanisms for knowledge editing remain poorly understood. This work employs Neuron-Level Knowledge Attribution (NLKA) to analyze the internal differences between successful and failed edits, revealing how attention and feed-forward networks jointly facilitate the injection of new knowledge and suppression of outdated information. For the first time, attribution results are translated into actionable engineering signals, leading to MEGA—a lightweight, architecture-agnostic activation intervention method that requires no weight modifications. MEGA explicitly identifies where and how edits take effect within the model and demonstrates superior performance over existing approaches on both GPT2-XL and LLaMA2-7B across the CounterFact and Popular datasets, enabling efficient and reliable knowledge updates.

This study investigates the internal representation mechanisms of factual knowledge in large language models (LLMs), introducing the concept of “knowledge circuits”—structured, cross-layer, cross-module computational pathways involving coordinated activation of attention heads and MLPs. Using computational graph tracing, module interaction analysis, knowledge-editing perturbations, and behavioral visualization, we systematically identify stable and reproducible factual knowledge propagation pathways in GPT-2 and TinyLLaMA for the first time. Our contributions are threefold: (1) advancing knowledge modeling from isolated parameters or modules to dynamic computational graphs; (2) enabling circuit-level, mechanistic explanations for hallucination and in-context learning; and (3) empirically demonstrating that prevailing knowledge editing methods disrupt these circuits, while also establishing circuit-aware interventions for diagnostic analysis and targeted correction.

This work addresses the susceptibility of large language models to catastrophic forgetting and model collapse during continual knowledge editing. The authors propose a mechanism-aware, precise editing framework that, for the first time, integrates sparse circuits with interpretable neurons. By leveraging a sparse transcoder to construct knowledge circuits, the method identifies and manipulates specific functional neurons, enabling fine-grained knowledge updates with minimal interference to unrelated model capabilities. Designed to support lifelong learning, the approach maintains strong performance on standard benchmarks such as MMLU and GSM8K even after 3,000 consecutive edits on Gemma2, Qwen3, and Llama3.1, significantly outperforming existing techniques.

Existing continual knowledge editing methods for large language models suffer from error accumulation due to parameter interference, degrading both editing accuracy and generalization. This paper proposes a fine-grained neuron localization framework coupled with an entropy-guided dynamic sparse masking mechanism. First, neurons are functionally attributed and categorized into *knowledge-general* and *knowledge-specific* types. Then, only critical knowledge-specific neurons undergo sparse, adaptive weight updates—minimizing parameter perturbation while preserving model integrity. The method requires no full model retraining and maintains high editing success rates (+12.7%) and strong generalization stability (38.5% reduction in forgetting) over thousands of sequential edits—outperforming state-of-the-art approaches. Its core innovations lie in (i) interpretable, function-based neuron partitioning and (ii) an information-theoretic, sparsity-aware editing strategy that balances fidelity and plasticity.

This study investigates the knowledge organization mechanisms underlying critical behavioral circuits in GPT-2. Addressing the challenges of locality and editability in language model knowledge representation, we propose a causal mediation–based knowledge editing framework that integrates ROME-based circuit localization, multi-layer probing, and attribution analysis to systematically evaluate editing response patterns, knowledge distribution breadth, and structural robustness. Our key contributions are: (1) the first empirical demonstration that factual knowledge is non-localized—distributed across layers and heads via collaborative semantic encoding; (2) evidence that single-point edits induce consistent semantic updates throughout the entire transformer stack; and (3) the finding that circuits exhibit both functional specificity and knowledge redundancy. Consequently, we formally define a circuit’s “meaning” as an intervenable, generalizable semantic functional unit—establishing a novel paradigm for model interpretability and safe, controllable knowledge editing.

This work addresses the degradation in output coherence and safety commonly induced by existing activation intervention methods that modulate large language model personas through perturbations of residual streams. To overcome this limitation, the authors propose a novel geometric analysis–based approach that leverages inter-layer cosine similarity and attention head contribution scores to identify—and for the first time explicitly name—three sparse, independent “style-modulating heads” responsible for disentangled control of persona and stylistic attributes. By precisely intervening on these specific attention heads, the method achieves effective persona modulation while substantially mitigating the coherence deterioration typical of conventional techniques, thereby significantly enhancing the robustness and safety of behavioral interventions in large language models.

Existing 3D editing methods often rely on category-specific models, limiting their generalizability. This work proposes a category-agnostic framework for versatile 3D shape editing based on Coupled Neural Shapes (CNS), which integrates a global semantic latent code with a 3D neural feature volume to enable diverse editing operations through joint optimization. The approach introduces two key innovations: a KV-cache replacement mechanism and latent feature regularization, which together allow precise control over edited regions while preserving geometric consistency in non-edited areas. Compatible with various 3D generative models, the method achieves state-of-the-art quantitative and qualitative results across multiple benchmarks, significantly enhancing both editing flexibility and shape fidelity.

Existing neural network editing methods rely on task-specific handcrafted algorithms, which are costly and exhibit poor generalization. This work proposes a unified, learnable framework by formulating model editing as a reinforcement learning problem for the first time. An agent learns to edit model parameters autonomously within two environments—MaskWorld (multiplicative mask scaling) and ShiftWorld (additive weight shifting)—guided by a multi-objective reward function that balances task-specific objectives with overall model performance preservation. Experiments demonstrate that the approach effectively reduces accuracy on forget sets to nearly 0% while maintaining over 90% accuracy on retain sets in machine unlearning tasks. In bias mitigation scenarios, it improves fairness metrics by more than 5% without compromising classification utility.

This work proposes a biologically inspired, unsupervised personalization method that eliminates the need for backpropagation and labeled data, addressing the inefficiency of conventional deep neural networks in resource-constrained settings. Drawing inspiration from neural pruning mechanisms in the brain, the approach introduces a fine-grained, plasticity-driven pruning strategy into mainstream architectures such as ResNet-50 for the first time. Evaluated on benchmarks including ImageNet, the method achieves approximately 70% sparsity while boosting accuracy to around 90%, substantially reducing computational overhead. This demonstrates a notable departure from the typical trade-off between model efficiency and performance, as the technique simultaneously lowers resource consumption and enhances predictive accuracy.

This work proposes a biologically inspired approach to generate sparse, modular recurrent neural networks from highly compressed genotypic encodings for efficiently solving complex temporal tasks. By integrating genotypic compression with modular reservoir computing for the first time, the method employs a hypernetwork to learn a structured generative process and introduces a dual developmental-evolutionary learning paradigm augmented with curriculum-based meta-learning. The resulting networks exhibit strong modularity and sparsity, achieving high performance and robustness with minimal task-specific training. Empirical evaluations demonstrate that the generated architectures significantly outperform existing methods across multiple challenging temporal benchmarks, highlighting the efficacy of combining developmental principles with evolutionary and meta-learning strategies in neural architecture design.