compute matrix derivatives

Efficient and accurate computation of higher-order derivatives of generalized eigenvalues/vectors for symmetric matrices and generalized singular values/vectors for rectangular matrices—parameterized linearly or nonlinearly—is essential yet theoretically underdeveloped, especially under degeneracy and practical constraints. Method: We rigorously derive closed-form expressions for first- and second-order analytical derivatives using matrix differential calculus, validate Jacobian and Hessian matrices via numerical differentiation, and implement an open-source R package supporting diverse degenerate cases and application-specific constraints. Contribution/Results: This work establishes the first unified, complete high-order differential theory framework for these spectral decompositions. The proposed method substantially improves both computational efficiency and numerical accuracy of derivative evaluation. It has been successfully applied to multivariate statistical modeling and covariance structure optimization, providing a rigorous theoretical foundation and practical computational infrastructure for parameter sensitivity analysis and gradient-based optimization in eigen-decomposition–dependent models.

This work addresses the gap between abstract Riemannian geometry and practical algorithmic implementation by systematically developing a computationally tractable geometric framework for Riemannian optimization. Focusing on canonical matrix manifolds—Stiefel, Grassmann, and symmetric positive-definite (SPD) manifolds—it explicitly derives core geometric structures, including tangent spaces, metric tensors, Levi-Civita connections, curvature operators, and geodesics, all expressed in coordinate- and matrix-based forms amenable to numerical computation. Furthermore, it provides closed-form expressions for the Riemannian gradient, Hessian, exponential map, and retraction operators. To the best of our knowledge, this is the first unified formulation that translates classical differential-geometric constructions into a consistent, implementation-ready framework, thereby bridging theory and practice and offering a rigorous foundation for efficient and accurate algorithm design in Riemannian optimization and geometric machine learning.

In machine learning, the prohibitively high computational cost of computing Jacobian and Hessian matrices severely limits their practical deployment in large-scale problems—especially when sparsity structure remains underutilized. To address this, we propose Automated Sparse Differentiation (ASD), a fully automatic sparse automatic differentiation framework. ASD introduces a novel operator-overloading mechanism that jointly detects global and local sparsity patterns, circumventing failures of control-flow-graph-based analysis. It implements a decoupled Julia software stack, compatible with any AD backend without requiring modifications to user code. The framework integrates sparse structural analysis, matrix coloring, and hybrid symbolic-numerical differentiation. Experimental results demonstrate that ASD achieves up to 1000× speedup on scientific machine learning and optimization tasks. Crucially, it enables efficient, single-pass generation of large-scale Jacobians and Hessians for the first time—outperforming conventional AD methods in both efficiency and scalability.

Automatic differentiation (AD) in deep learning and scientific computing suffers from poor structure preservation and inefficient high-order derivative computation. Method: This paper establishes a geometric framework grounded in jet bundles and Weil algebras: reverse-mode AD is interpreted as cotangent pullback, while higher-order Taylor expansions correspond to algebraic evaluation over Weil algebras. Contribution/Results: We introduce tensorized Weil algebras, enabling simultaneous computation of all mixed partial derivatives and circumventing combinatorial explosion from nested Jacobian-vector or vector-Jacobian products. For the first time, correctness and numerical stability of AD are rigorously guaranteed via functorial identities and algebraic exactness. The algorithm exhibits linear complexity in the algebraic dimension and provides explicit bounds on truncation error, thereby establishing a unified theoretical foundation for structure-preserving differential methods.

This work addresses the efficient and robust computation of gradients for numerical solutions of differential equations. We systematically survey four differentiable programming paradigms—adjoint methods, automatic differentiation (via source-to-source transformation and operator overloading), numerical perturbation, and symbolic-numeric hybrid approaches—and introduce, for the first time, a unified differentiability framework that bridges inverse problem solving and machine learning methodologies. We establish a cross-method comparative taxonomy and provide platform-specific best-practice guidelines for scientific computing libraries including SciPy, JAX, and TorchDiffeq. Our analysis rigorously characterizes trade-offs among accuracy, memory footprint, computational complexity, and applicability domains for each method. The results deliver both theoretical foundations and practical implementation pathways for differential-equation–data fusion modeling tasks, including parameter inversion, sensitivity analysis, and physics-informed neural networks (PINNs).

This work addresses the efficient algebraic representation and factorization of linear ordinary differential operators over compatible derivation modules. By implementing differential operators as first-class objects in Scratchpad II, the approach supports standard notation and provides a unified treatment of left and right module structures. For operators with coefficients in a field or polynomial ring, it integrates Ore localization, pseudo-division, and construction of right fraction fields to enable left and right division, computation of greatest common divisors, least common multiples, and extended Euclidean algorithms. Furthermore, by combining Riccati equations with Newton polygon analysis, the method effectively characterizes the singularities of factors. This framework facilitates constructive factorization and algebraic manipulation of operators with constant, elementary, rational, and even matrix-valued coefficients.

Existing methods struggle to compute high-order derivatives—such as the Hessian and observed information matrix—of SE(3) objective functions efficiently and accurately, hindering rapid algorithm prototyping. This work proposes a hybrid approach that integrates analytical differentiation with vector-seed dual-number automatic differentiation, partitioning the computation at point-action interfaces to enable exact arbitrary-order derivatives from a single gradient implementation. The key innovation lies in the first-time combination of analytical Lie group Jacobians with dual-number AD, allowing exact Hessian computation via a single forward pass while effectively removing the removable singularity of SO(3)/SE(3) bases at the origin. Experiments on a canonical 6-DoF, 5-landmark problem demonstrate that the method is approximately five times faster than finite differences, achieves machine-precision accuracy, requires only about 70 additional lines of analytical code, and involves no hyperparameter tuning.

This work addresses the inefficiency and numerical instability of existing trajectory optimization methods, which typically rely on numerical or automatic differentiation to compute Jacobians of high-order time derivatives—such as jerk and rate of force change—while neglecting the structural properties of multibody systems. The authors propose a novel analytical framework that explicitly models physical quantities and their higher-order derivatives by leveraging the inherent structure of multibody dynamics. For the first time, they derive structured Jacobian matrices with respect to generalized coordinates and their higher-order time derivatives. By integrating analytical differentiation with multibody dynamics, the method enables efficient and scalable forward and inverse optimization. It significantly improves computational efficiency and numerical stability compared to conventional approaches and demonstrates success in accurately recovering cost function weights from motion data in inverse optimization tasks.

Numerical differentiation is essential in scientific computing and engineering, yet robust and accurate derivative estimation remains challenging for noisy, undersampled, or non-stationary data. To address this, we systematically survey state-of-the-art numerical differentiation methods under noise and propose the first unified classification framework integrating boundary handling, regularization mechanisms, and adaptivity to frequency- or time-domain characteristics. We introduce a data-quality–aware algorithm selection criterion based on noise level, sampling rate, and non-stationarity. Our framework unifies twelve method classes—including finite differences, Tikhonov and total variation regularization, Savitzky–Golay filtering, spectral methods, and Kalman smoothing—and is implemented in the open-source Python library PyNumDiff, supporting five automated hyperparameter strategies. Extensive experiments demonstrate substantial improvements in estimation accuracy and robustness. The framework is validated on real-world applications in physical modeling and biosignal analysis.