orthogonal rotations

This work addresses the challenge of efficiently fine-tuning 3D foundation models for downstream tasks under complex variations in texture, geometry, camera motion, and illumination. The authors construct a controllable synthetic dataset to independently train LoRA adapters and extract low-rank subspaces corresponding to each attribute, providing the first empirical validation that these subspaces are approximately orthogonal and disentangled in 3D data. Building on this finding, they fuse multiple attribute-specific subspaces into a compact, shared LoRA subspace, which—despite being trained solely on synthetic data—demonstrates strong generalization to real-world scenes. The proposed method significantly enhances downstream task performance while maintaining high parameter efficiency, establishing a novel paradigm for effective transfer learning in 3D vision.

This study investigates why singular value decomposition (SVD)-based orthogonalization degrades SO(3) rotation estimation performance during training, despite SVD outperforming the Gram–Schmidt method at inference. By deriving, for the first time, the exact spectral structure of the Jacobian in SVD backpropagation, the authors reveal that small singular values induce severe gradient distortion during training. They further demonstrate that the 6D Gram–Schmidt parameterization suffers from imbalanced gradient flow, providing theoretical support for the superiority of 9D representations. Based on these insights, the work proposes a paradigm that avoids orthogonalization during training and applies SVD only at inference, thereby establishing both theoretical grounding and practical guidance for 9D regression followed by SVD-based projection.

This work addresses the significant performance degradation of existing deep image matching methods under large in-plane rotations. Through systematic investigation of where to best incorporate rotation invariance within sparse feature matching pipelines, extensive training, and multi-benchmark evaluation, the study demonstrates that introducing rotation invariance solely at the descriptor stage achieves robustness comparable to that of rotation-invariant matchers while being more computationally efficient. Moreover, it shows that, with sufficient training data, rotation invariance does not compromise general matching performance and highlights the critical role of data scale in enabling robust rotation generalization. The released models achieve state-of-the-art results on benchmarks including WxBS, HardMatch, and SatAst, substantially improving matching robustness across multimodal, extreme-viewpoint, and satellite imagery scenarios.

This work addresses the degradation of regression performance under in-plane image rotations by proposing a geometry-aware approach based on Convolutional Model Trees (CMT). The method enhances rotational invariance during training through three geometric inductive biases: convolutional smoothing, tilt-dominance constraints, and importance-based pruning. Furthermore, it introduces, for the first time in model tree ensembles, a deployment-time strategy that combines discrete rotation search with a confidence-proxy optimization mechanism to adaptively select the optimal rotation orientation. Experimental results demonstrate that the proposed approach significantly improves robustness under large-angle rotations on the MNIST rotation regression task, although a slight performance drop may occur near the canonical orientation due to misalignment between confidence estimates and prediction accuracy.

To address weak class discriminability and insufficient feature robustness in machine learning, this paper proposes a Multi-level Orthogonal Subspace (MOS) Karhunen–Loève feature theory within a random tensor space. Training data are modeled as stochastic processes in a Bochner space, and hierarchical KL expansions explicitly decouple dominant class structures from inter-class anomalous signals, enabling class-wise subspace disentanglement and interpretable projection features. This work establishes, for the first time, a MOS feature construction paradigm under the random tensor framework—uniquely integrating statistical modeling rigor with geometric interpretability. Evaluated on the ADNI plasma dataset, the method significantly outperforms gradient boosting, RUS Boost, random forests, and CNNs, achieving substantial gains in classification accuracy. These results validate its robust discriminative capability for high-noise biomedical data.

Existing SVD-based low-rank adaptation methods overlook the coupled geometric structure between the left and right singular bases of pre-trained weights, limiting their stability and parameter efficiency. This work proposes a slice-wise consistent orthogonal rotation mechanism that, after SVD truncation, applies a shared orthogonal transformation and diagonal spectral shift to the left and right singular bases across all row slices. This approach explicitly models their geometric coupling while inducing minimal perturbation to the original weights. Notably, it achieves, for the first time, slice-level consistent orthogonal transformations, substantially reducing the number of trainable parameters. Experimental results demonstrate that the proposed method outperforms state-of-the-art approaches—including LoRA, DoRA, PiSSA, and MiLoRA—on commonsense reasoning and code generation tasks, using only approximately one-eighth of their trainable parameters.

This work addresses the inefficiency in spectral utilization within the low-rank subspace of trained LoRA adapters, where many singular directions are either unhelpful or detrimental to downstream tasks. The authors propose the first training-free post-processing method for LoRA: by performing SVD on the trained LoRA weights and estimating the sensitivity of each singular component via gradients computed on a small calibration set, they reweight the singular values according to their sensitivity while preserving the original singular directions. This approach adjusts only around 1,000 scalar coefficients and yields significant performance gains across four benchmarks on Llama-3.1-8B and Qwen3-8B, achieving up to a 4.4-point improvement on CommonsenseQA and a 2.4-point gain in HumanEval pass@1.

This work addresses the trade-off between performance and efficiency in parameter-efficient fine-tuning of large language models by systematically reinterpreting Low-Rank Adaptation (LoRA) through the lens of signal processing. Leveraging classical low-rank modeling and inverse problem theory, it establishes a unified framework to understand both existing and future efficient fine-tuning methods. The study proposes a three-dimensional technical framework encompassing architecture design, optimization strategies, and full-lifecycle deployment, integrating core techniques such as singular value decomposition, rank expansion, cross-layer tensorization, norm-invariant optimization, and parameterization-aware solvers. This approach provides theoretical grounding and principled design guidelines for LoRA and its variants, while extending their applicability across pre-training, post-training, and deployment stages, thereby fostering bidirectional integration between signal processing and deep learning.

Existing parameter-efficient fine-tuning methods struggle to preserve the geometric structure of pre-trained model representations under low-rank updates. This work proposes a novel fine-tuning approach that constructs orthogonal transformations using low-rank skew-symmetric matrices and introduces parallelizable composition chains of rotations. By doing so, it simultaneously ensures orthogonality, computational efficiency, and controllable approximation error in high-dimensional spaces. The method uniquely integrates low-rank compositional rotations with orthogonal constraints, achieving state-of-the-art or competitive performance across diverse tasks—including diffusion Transformers, vision Transformers, and language model fine-tuning—outperforming both existing orthogonal and non-orthogonal approaches.

This work addresses the challenge of effectively fusing orthogonally fine-tuned adapters—each specialized for distinct concept and style tasks—without requiring additional training, to enable high-quality multi-attribute image generation. Leveraging the Riemannian manifold structure of Group-and-Shuffle orthogonal matrices, the authors propose a training-free multiplicative adapter fusion method. This approach employs an efficiently approximated geodesic interpolation formula on the manifold and incorporates a spectral restoration transformation to preserve the spectral properties of the fused adapter. The method achieves, for the first time, joint generation using orthogonally fine-tuned adapters without further fine-tuning, successfully synthesizing high-fidelity images that simultaneously embody specified styles and concepts in subject-driven generation tasks, thereby demonstrating its effectiveness and superiority.