orthogonal regularization

This work addresses the challenge of entangled, superimposed, and poorly interpretable feature representations in language models by proposing an orthogonality-regularized fine-tuning approach for sparse autoencoders. Specifically, the decoder matrix is decomposed into an approximately orthogonal set of feature bases, grounded in the principle of independent causal mechanisms. The orthogonality constraint enhances feature uniqueness and modularity, while feature identifiability and interpretability are further improved through embedding distance metrics and causal intervention analysis. This enables more precise and isolated manipulation of individual features. Experimental results demonstrate that the proposed method yields substantially clearer and more controllable internal representations, with negligible degradation in downstream task performance.

This study investigates whether geometric regularization—specifically, orthogonality-inducing losses in weight space—effectively promotes diversity and specialization among experts in Mixture-of-Experts (MoE) models. Through systematic ablation experiments across multiple datasets (WikiText-103, TinyStories, PTB), the authors apply varying strengths of orthogonality loss and evaluate its impact using metrics of weight and activation overlap, such as Mean Squared Overlap (MSO). The results reveal that orthogonality regularization neither consistently reduces weight or activation redundancy nor yields reliable performance improvements; in some cases, it even degrades model performance. Crucially, the study uncovers no significant correlation between weight-space orthogonality and expert activation behavior (r = –0.293, p = 0.523), thereby challenging the prevailing assumption that geometric regularization in weight space effectively enhances expert specialization in MoE architectures.

This work investigates the statistical–computational trade-off of the Nyström method under general convex Lipschitz loss functions (e.g., hinge loss), specifically addressing whether randomized subspace approximation compromises learning accuracy. It extends Nyström analysis to nonsmooth losses for the first time, establishing a unified theoretical framework that enables rigorous translation from surrogate risk bounds to classification error bounds—thereby facilitating comparable performance analysis across settings such as hinge and squared loss. Theoretically, under mild assumptions on kernels and data distributions, the method achieves optimal generalization error while substantially reducing computational complexity. It recovers classical results for smooth losses and, crucially, delivers tight, verifiable error bounds for practical classification tasks—including SVM—along with empirically grounded guidance for algorithmic design and implementation.

This study addresses the limited generalization performance of standard norm-based regularization in neural networks when dealing with high-dimensional or feature-correlated settings, where conventional methods inadequately control model complexity. To overcome this, the authors propose two covariance-aware adaptive regularization techniques: first, incorporating the input feature covariance structure into ℓ₂ weight decay to refine ridge-type penalties; second, combining ℓ₁ sparsity with covariance-informed ℓ₂ regularization to achieve structured sparsity. By innovatively embedding feature covariance information into classical Lasso and ridge regression frameworks, the approach enables more precise complexity control. Extensive experiments on Monte Carlo simulations and real-world datasets—including building cooling load prediction and leukemia cell classification—demonstrate that the proposed methods significantly outperform traditional regularization strategies and substantially enhance model generalization.

This work addresses the Sparse-Constrained Orthogonal Non-negative Matrix Factorization (SCONMF) problem—approximating a data matrix via low-rank factorization under three constraints: non-negativity of both factor matrices, row-wise orthogonality of the mixing matrix, and an upper bound on the number of nonzeros per row. We formulate SCONMF as a Capacity-Constrained Facility Location Problem (CCFLP) for the first time, and propose a novel constrained optimization framework integrating Control Barrier Functions (CBFs) with the maximum entropy principle to rigorously enforce all constraints. Moreover, we introduce the first quantifiable “true rank” criterion for evaluating intrinsic rank. Experiments on multiple benchmark datasets demonstrate that our method reduces reconstruction error by up to 150× compared to state-of-the-art approaches, substantially improving both approximation accuracy and structural interpretability of the decomposition.

This work investigates whether joint multi-task modeling under orthogonality constraints offers a parameter complexity advantage over independent task modeling. By constructing an orthogonal function class composed of shared hard features and task-specific readout heads—leveraging tools such as Rademacher–Haar wavelets, Sawtooth–Walsh readout functions, and Heaviside-activated networks—the study provides the first information-theoretic proof that, under orthogonality, the description length of a joint representation is significantly shorter than that of separate representations. The theoretical analysis establishes tight upper and lower bounds for both approximation regimes, revealing the critical role of shared latent features in reducing the number of encoding bits and thereby demonstrating the expressive efficiency of compositional multi-output architectures.

This study addresses the limitation of conventional research that reduces neural network interference to geometric overlap while neglecting code statistics and actual interactions. We propose the concept of "effective interference," which integrates feature geometry with coding statistics to distinguish constructive from destructive interference and quantify interaction strength. Based on sparse autoencoders and a local fixed-support assumption, this work reveals that constrained architectures shape interference patterns through four mechanisms, including orthogonalization and bias compensation. Our findings demonstrate that architectural constraints selectively reduce co-activation overlap while preserving beneficial cross-contributions. These results establish that interference inherently depends on usage patterns and network architecture, thereby overcoming the theoretical limitations of purely geometric perspectives.

本文探讨了多类学习中算法原则的局限性，通过证明某些问题无法通过适当学习或正则化解决，并给出SRM可学习性的充分条件。

This study addresses the challenge of selecting an optimal regularization method that balances predictive accuracy and feature selection stability based on data characteristics. Through systematic Monte Carlo simulations across a seven-dimensional parameter space—encompassing 134,400 experiments with eight production-grade models—the authors evaluate Ridge, Lasso, ElasticNet, and Post-Lasso OLS. They reveal, for the first time, that Lasso suffers severe recall degradation (as low as 0.18) under conditions of high multicollinearity and low signal-to-noise ratio (SNR), whereas ElasticNet remains robust (achieving a recall of 0.93). The work proposes practical selection guidelines based on sample size, feature correlation, and SNR, and demonstrates that when the sample-to-feature ratio is sufficiently large (n/p ≥ 78), mainstream methods exhibit comparable predictive performance.

This work addresses the limitation of conventional element-wise reconstruction error in tensor low-rank approximation, which fails to capture geometric degradation of multidimensional structures. Building upon the orthogonal Tucker model, the paper introduces a novel orthogonal decomposition of reconstruction error into directional loss—quantifying subspace deviations caused by truncation and noise—and interaction loss—measuring distortions in the multilinear interactions of the core tensor. A Wedin-type stability bound is established for the directional loss. Experiments on synthetic and hyperspectral data, leveraging matrix SVD, tensor Tucker decomposition, and subspace perturbation analysis, demonstrate that under identical reconstruction errors, directional loss can vary by up to 4.6×, and high directional loss strongly correlates with visual blurring, thereby underscoring the necessity of structure-aware error metrics.