parametric memory decoding

This work addresses the limitations of existing LoRA-based external parameter memory (EPM) routing methods, which rely on additional training and lack a unified benchmark and systematic design for zero-shot routing. We propose the Parameter Memory Decoding (PMD) framework, reframing zero-shot LoRA routing as a decoding process of EPM activations, and introduce PMD-Bench—the first comprehensive benchmark for evaluating routing performance. Our PMDRouter requires no extra routing modules; instead, it leverages the magnitude of responses from a single prefill pass of the base model to efficiently score and route among parameter memories. Experiments demonstrate that PMDRouter achieves state-of-the-art internal signal routing performance across diverse settings in PMD-Bench, confirming the feasibility of zero-shot LoRA routing and highlighting the general applicability and improvement potential of the PMD framework.

Existing parameter reparameterization methods are often confined to a single objective—either parameter-efficient fine-tuning or model compression—making it challenging to simultaneously address both demands under resource constraints. This work proposes CRISP, a unified framework that jointly achieves model compression and parameter-efficient fine-tuning within a single architecture. CRISP decomposes pre-trained weights into shared base matrices and lightweight mixture coefficients, enhanced by cross-layer base sharing and an interpolation-based gated coefficient recombination mechanism. Requiring fewer than 200 trainable parameters, CRISP outperforms existing approaches by 1% in joint compression and fine-tuning tasks, surpasses state-of-the-art methods by up to 1.5% in pure parameter-efficient fine-tuning, and achieves a consistent 4–5% improvement in overall dual-task performance.

This work addresses the challenge that large language models struggle to effectively follow textual skill instructions in long-context scenarios, which hinders practical skill deployment. To overcome this limitation, the authors propose ParametricSkills, a novel framework that, for the first time, dynamically converts free-form textual skills into LoRA adapter parameters at test time, enabling context-independent skill invocation and establishing a new paradigm for test-time continual learning. Leveraging a large-scale skill repository and invocation trajectories derived from OpenCode, a hypernetwork is trained to map textual descriptions to parameterized skills. Evaluated across six software engineering subtasks, the method outperforms in-context learning by an average of 6.44 points under DeepSeek-V4-Flash assessment, with significant improvements in both BERTScore and F1 metrics.

Neural network interpretability lacks formal, mechanism-grounded parameter decomposition that is both input-adaptive and faithful to underlying computational structures. Method: We propose Attribution-based Parameter Decomposition (APD), the first method to perform mechanism-driven parameter decomposition directly in parameter space. APD integrates attribution-guided differentiable modeling, mechanism sparsity regularization, and description-length minimization to decompose parameters into concise, input-adaptive, and faithful mechanistic components. Contribution/Results: APD provides a formal conceptual foundation for “features” in deep networks, enabling identification of cross-layer distributed representations and hyper-localized mechanisms. In controlled experiments, APD successfully recovers hyper-localized features, disentangles compressive computations, and localizes cross-layer distributed mechanisms. It establishes a novel paradigm for minimal-circuit discovery and architecture-agnostic parameter decomposition, advancing mechanistic interpretability beyond post-hoc attribution.

The theoretical mechanisms underlying parameter-efficient fine-tuning (PEFT) methods for large pre-trained models remain poorly understood, and the performance disparities among existing approaches lack principled explanations. Method: This paper establishes, for the first time, a unified theoretical framework grounded in matrix decomposition, revealing that diverse PEFT methods fundamentally perform optimization under low-rank constraints. Leveraging this insight, we propose two novel PEFT methods and a general-purpose enhancement framework—designed with theoretical rigor and architectural generality—through SVD- and LoRA-style modeling analysis, modular design, and multi-task empirical validation. Contribution/Results: Our approach significantly improves the performance of canonical PEFT methods—including LoRA and Adapter—across mainstream NLP benchmarks. This work provides the first principle-level, systematic explanation of PEFT and establishes an extensible technical pathway for future advancements.

Existing interpretability methods struggle to distinguish whether model components genuinely encode a target capability or merely propagate upstream signals. This work proposes Weight Patching, a source-directed intervention in weight space that operates on isomorphic models exhibiting varying behavioral strengths. By substituting specific module weights and anchoring behavioral interfaces via vector alignment, the method precisely localizes source-level mechanisms within large language models. The framework enables, for the first time, tracing the pathway of capability transmission from shallow source carriers to downstream execution circuits, thereby supporting mechanism-aware model merging. Experiments on instruction-following tasks successfully identify critical mechanistic components, significantly improving selective fusion of expert models, with findings further validated externally.

This study investigates whether explicitly exposing routing mechanisms in Transformers is sufficient to achieve mechanistic interpretability. To this end, the authors propose Block Attention Residuals, which represent cross-layer information routing as observable tensors during forward propagation and enable causal intervention to analyze their functional roles. Experiments based on the Qwen3 architecture demonstrate that meaningful local routing patterns emerge only when the routing structure is actively involved in training optimization. Crucially, the magnitude of routing weights does not directly reflect causal importance, necessitating intervention-based validation of interpretability hypotheses. The work identifies three characteristic local routing patterns and reveals that segments with the largest routing weights do not necessarily contribute the most causally, establishing that explicit exposure of routing mechanisms is necessary but insufficient for mechanistic interpretability.

This work clarifies common misconceptions that dynamic parameterization inherently enables dynamic inference or computational savings, explicitly distinguishing among coefficient variation, dependence of frozen models on coefficient assignment, and conditional execution. To this end, the authors propose Frozen Controller Audit (FCA), a method that caches coefficient tensors and replays frozen models under various strategies—including cross-input reallocation, token shuffling, and static configurations—to quantify a model’s reliance on content-conditional coefficient assignment. Experiments on FeatureGate Transformers and MUDDPythia reveal that static configurations retain over 98.7% of original performance, while layer identity accounts for 87%–96% of coefficient variance. Notably, dynamic parameterization fails to accelerate inference—actually slowing it by 30.8%—and reallocation substantially increases negative log-likelihood, demonstrating strong dependence on content-conditional assignment without tangible efficiency gains.

This study addresses the limitations of traditional unidimensional factor models, which rely on correlation matrices and fail to capture their actual explanatory and predictive power for the original response matrix. The authors propose Refactor analysis, which translates a unidimensional solution into a rank-1 approximation of the raw response matrix via dual association plots, and integrates Verifactor analysis with row–column double cross-validation to enhance generalization. For the first time, unidimensionality assessment is shifted from fitting correlation matrices to directly predicting observed data, exposing a disconnect between conventional fit indices and data recoverability. The quadrant correlation coefficient (q′) is reintroduced as a robust alternative. Experiments on 200 public binary datasets demonstrate that q′ substantially outperforms standard correlation-based methods in reconstruction accuracy and sample stability, whereas traditional fit indices—despite high intercorrelations—prove poor predictors of actual recovery performance.

This work addresses the attention misalignment and error accumulation in foundational segmentation models during closed-loop iterative prompting, which stem from decoder coupling drift. The study is the first to formally characterize this phenomenon by modeling iterative prompting as a discrete-time dynamical system. Building upon the SAM architecture, the authors propose a training-free inference-time stabilization framework that leverages truth-agnostic prompt–image coupling metrics, attention stability analysis, and a proximal anchoring strategy to constrain prompt updates across iterations, thereby preserving decoder coupling consistency. Experiments on volumetric electron microscopy data demonstrate that the proposed method significantly enhances attention stability, temporal consistency, and segmentation accuracy, outperforming existing iterative prompting approaches.