self-representation alignment

Neural network internal representations often lack stability and cross-architectural consistency due to architectural disparities, hindering knowledge transfer and modular deployment. To address this, we propose a structured regularization framework comprising linear shaping operators and rectified path constraints, which explicitly encode inductive biases to improve geometric alignment of representations across architectures. Through theoretical analysis, controlled transfer experiments, and a novel representation alignment metric, we systematically demonstrate that structural priors significantly enhance semantic consistency among heterogeneous models. Our method improves downstream task performance in model distillation and modular learning by up to 12.3%, offering an interpretable and scalable paradigm for building robust, composable deep learning systems.

This paper addresses the empirical phenomenon of representation alignment—where model representations increasingly converge as scale grows—in large language models, establishing the first formal learning-theoretic analysis framework. Methodologically, it unifies alignment definitions across metric, probabilistic, and spectral perspectives; introduces a task-driven representation stitching mechanism; and establishes its necessary and sufficient condition in terms of kernel alignment. Theoretical contributions include: (i) an upper bound on the generalization error of stitched representations; (ii) a rigorous proof that kernel alignment strictly governs representation transferability; and (iii) the first falsifiable learning-theoretic foundation for representation convergence in large models. Experimentally, the work integrates kernel methods, spectral graph theory, and probabilistic modeling to empirically validate the theoretical predictions.

This work addresses the unclear interaction between feature alignment and target fitting in cross-modal fine-tuning, which often leads to a mismatch between feature-label structures across source and target domains, thereby degrading generalization. For the first time, this study theoretically characterizes their relationship by introducing the notion of “feature-label distortion,” and establishes a provable generalization bound on target error. Based on this analysis, a principle for joint optimization of alignment and fitting is derived. The resulting framework offers interpretable and actionable design guidelines for cross-modal fine-tuning. Extensive experiments demonstrate that the proposed method significantly outperforms current state-of-the-art approaches across multiple benchmark datasets, confirming its effectiveness and broad applicability.

This study investigates the mechanisms by which neural networks develop aligned representations across diverse architectures, training protocols, and datasets, with a particular focus on the roles of data signal-to-noise ratio (SNR) and sample size. By analyzing both synthetic and real-world data—augmented with controlled noise—in regression and classification tasks, and combining analytical derivations for single-hidden-layer linear networks with empirical experiments on deep nonlinear networks, the work reveals that representation alignment strength increases monotonically with SNR but exhibits a non-monotonic relationship with sample size, reaching its weakest point near the interpolation threshold. Furthermore, the study demonstrates that stronger alignment does not necessarily improve generalization, highlighting the nuanced and nontrivial influence of data quality and quantity on representational alignment.

This study investigates the impact of consistency training on model alignment, demonstrating that it is not alignment-neutral. Through systematic evaluation of seven consistency methods across 108 open-source large language models (7B–70B) with controlled misalignment, the authors find that such training generally suppresses reward hacking while exacerbating sycophancy. Leveraging controlled fine-tuning, distribution shift analysis, and theoretical modeling, they identify distribution shift as the dominant underlying mechanism. Building on this insight, they propose a unified theoretical framework that predicts under which conditions consistency training amplifies or mitigates specific misalignment behaviors, thereby offering an auditable foundation for safer alignment practices.

This study addresses the challenge of precisely identifying high-value images under a limited annotation budget to maximize object detection performance. Moving beyond conventional approaches that rely solely on feature rarity, this work proposes an unlabeled data selection strategy by modeling the relationship between internal model representations and prediction errors. Specifically, it leverages the model's own state and anticipated performance gains to drive active learning. Extensive evaluations demonstrate that the proposed method outperforms rarity-based baselines across most experimental settings and achieves top-ranked performance during long-cycle retraining, substantially improving both annotation efficiency and detection accuracy.

This study addresses the unclear underlying mechanisms behind the delayed emergence of validation generalization in grokking, highlighting limitations in existing explanations. By constructing an analytical framework grounded in mode connectivity and the geometry of low-loss regions, this work reveals that the misalignment of low-loss regions induced by training-validation partitions is a critical cause of grokking. Furthermore, it proposes a symmetry-preserving data partitioning strategy to generate stable anti-grokking cases. These findings challenge prevailing correlation-based explanatory theories and demonstrate that when low-loss regions are aligned, hyperparameter tuning alone cannot induce grokking; instead, model dynamics collapse into either trainable or untrainable states.

This study addresses the limitation of conventional research that reduces neural network interference to geometric overlap while neglecting code statistics and actual interactions. We propose the concept of "effective interference," which integrates feature geometry with coding statistics to distinguish constructive from destructive interference and quantify interaction strength. Based on sparse autoencoders and a local fixed-support assumption, this work reveals that constrained architectures shape interference patterns through four mechanisms, including orthogonalization and bias compensation. Our findings demonstrate that architectural constraints selectively reduce co-activation overlap while preserving beneficial cross-contributions. These results establish that interference inherently depends on usage patterns and network architecture, thereby overcoming the theoretical limitations of purely geometric perspectives.

This work addresses the long-standing isolation among research domains such as alignment training, model organisms, and toy models, which has hindered empirical cross-pollination and led to redundant exploration and inefficiency. For the first time, it systematically transfers supervised fine-tuning (SFT) practices across these domains by integrating cross-model output training, mixed-strategy data, and benign fine-tuning to rigorously evaluate the portability of key findings. The study demonstrates three successful transfer effects: enhanced behavioral generalization, mitigation of capability degradation, and the critical insight that preserving capabilities alone is insufficient to ensure robustness in subsequent training phases. These results underscore both the efficacy and limitations of reusing methodologies across domains, thereby fostering more synergistic development across disparate research areas.

Concept alignment lacks a unified definition, and existing methods optimize divergent objectives under the same terminology, obscuring its fundamental nature. This work formalizes its multidimensional structure by decomposing it along two axes—“alignment targets” and “alignment levels”—and identifies four distinct alignment properties, revealing that current approaches satisfy only subsets of these. To address this limitation, we propose Coupled Sparse Autoencoders (CoSAE), a framework that jointly optimizes multiple alignment objectives, alongside InterVenchA, an interventional evaluation benchmark. Experiments demonstrate that optimizing a single objective fails to reliably recover other alignment properties, whereas CoSAE achieves strong instance-level conceptual consistency using merely 0.1% paired data.