autodiff implementation

Automatic differentiation (AD) in C++ scientific computing faces significant deployment challenges due to the language’s complexity and limited support for heterogeneous architectures—particularly GPUs. Method: This paper proposes a high-performance, cross-platform AD approach based on source-to-source transformation. It extends the Clad AD framework to support the Kokkos abstraction layer, enabling unified gradient generation and optimization across diverse GPU architectures—including NVIDIA H100, AMD MI250x, and Intel Ponte Vecchio—while preserving Kokkos’ portable programming model. Contribution/Results: The method automatically generates efficient reverse-mode gradient code without compromising portability. Experimental evaluation shows that gradient computation overhead remains bounded at ≤2.17× the original function’s execution time. This advancement substantially enhances both the practicality and performance portability of C++ in differentiable scientific simulation and AI-integrated workflows.

This work addresses the longstanding divide between program execution and gradient-based optimization, which has hindered the use of general-purpose code as learnable scientific models. The authors propose the Differentiable Meta-Circular Interpreter (DMCI), which compiles a self-hosted subset of Scheme into a differentiable computation graph, enabling exact reverse-mode automatic differentiation of arbitrary recursive, higher-order programs with closures and complex data structures—without requiring recompilation or custom gradient definitions. This approach facilitates joint optimization over both program structure and parameters. Validated on 171 programs for gradient correctness, DMCI significantly outperforms handcrafted models and gradient-free methods in inverse problems such as battery degradation and El Niño prediction, successfully achieving symbolic regression and extrapolation with stateful executable programs.

We propose a new method, that we coined the ``morphism-trick'', to integrate custom implementations of vector-Jacobian products in automatic differentiation softwares, applicable to a wide range of semiring-based computations. Our approach leads to efficient and semiring-agnostic implementations of the backward pass of dynamic programming algorithms. For the particular case of finite-state methods, we introduce an algorithm that computes and differentiates the $\oplus$-sum of all paths'weight of a finite-state automaton. Results show that, with minimal effort from the user, our novel library allows computing the gradient of a function w.r.t. to the weights of a finite state automaton orders of magnitude faster than state-of-the-art automatic differentiation systems. Implementations are made available via an open-source library distributed under a permissive license.

This work addresses the challenge of supporting task parallelism in reverse-mode automatic differentiation (AD) for functional languages—particularly Haskell—where conventional dual-number-based reverse AD struggles with parallel execution. We present the first parallel reverse AD implementation targeting the standard Haskell98 subset, requiring no explicit serialization of computation graphs. Our approach integrates dual-number representation, linear decomposition, and mutable array optimizations to enable task-level parallelism in derivative computation while preserving pure functional semantics. Crucially, a compile-time functional transformation retains the original program’s parallel structure, eliminating the sequential dependencies inherent in traditional reverse AD. Contributions include: (i) the first extension of dual-number reverse AD to task-parallel settings; (ii) theoretically optimal asymptotic complexity for both time and space; and (iii) empirical validation demonstrating that derivative computation achieves parallel efficiency and scalability equivalent to the original program.

Neural ODE training suffers from high computational cost, excessive memory consumption, and numerical instability during backpropagation. To address these challenges, this paper introduces the Algebraically Invertible ODE Solver family, grounded in algebraically invertible numerical integration. The method integrates high-order implicit/explicit reversible schemes, adjoint-state techniques, and memory–computation co-optimization to achieve, for the first time, high-order accuracy, strict numerical stability, and exact gradient computation in backpropagation. Unlike recursive checkpointing, our approach achieves strictly superior time and memory complexity bounds. Extensive evaluation on multiple benchmark ODE tasks demonstrates a 2.1× reduction in training latency and a 68% decrease in GPU memory usage, while preserving gradient precision and numerical robustness.

This work addresses the challenge of implementing reverse-mode automatic differentiation for programs featuring algebraic effects such as finite discrete probabilistic choice. Building upon the Compositional Homomorphic Automatic Differentiation (CHAD) framework, it formulates automatic differentiation as a semantics-preserving program transformation and constructs a backward-pass mechanism tailored to the finite atomic distribution monad. The correctness of this construction is established using logical relations from category theory. This study presents the first systematic extension of reverse-mode automatic differentiation to effectful languages with discrete outputs, introducing a reusable differentiation scheme applicable to a broad class of algebraic effects—including nondeterminism, exceptions, and writer effects. The approach not only enables correct reverse differentiation of programs with finite discrete probabilistic structure but also lays a foundational theoretical groundwork for differentiating more general effectful languages.

This work addresses the challenge of deploying deep neural networks with massive parameter counts by proposing a structured weight generation method based on differentiable nonlinear tensor networks. Leveraging tree tensor networks (TTN), augmented TTN (aTTN), and multiscale entanglement renormalization ansatz (MERA), the approach integrates nonlinear activations and task-aware objectives to end-to-end train compact core tensors that implicitly generate full-weight matrices. The architecture enables hardware-aware tensor contraction scheduling, achieving per-layer compression ratios ranging from 2,000× to 77,000× on AlexNet and VGG-16 while matching or even surpassing the accuracy of the original dense models, thereby substantially improving model compressibility and deployment efficiency.

Training large language models with conventional backpropagation requires caching gradients for all layers, incurring substantial memory overhead. This work proposes integrating optimizer updates directly into the backward pass through register-level operator fusion and tiling strategies, enabling immediate consumption and application of gradients upon their generation—thereby avoiding explicit instantiation of full gradient tensors. The method preserves mathematical equivalence in full precision, supports tensor and sequence parallelism, accommodates diverse optimizers, and extends to bf16 and 8-bit low-precision training. Experiments demonstrate over 50% reduction in optimizer memory usage, a 1.5× speedup in small-batch training, and the ability to scale micro-batch sizes by 4× for training an 8B-parameter model on the same GPU hardware.

Existing approaches lack the capability to perform automated backward error analysis for numerical programs, making it difficult to verify their backward stability. This work proposes a formal framework that generalizes the definition of backward stability, introduces the category Shel to model stable numerical computations, and develops the tool eggshel to automatically synthesize error bounds. The framework incorporates a novel, composable, and flexible notion of stability, integrating category theory, formal verification, and symbolic reasoning to automatically search for stability proofs within subcategories of Shel. Notably, eggshel is the first tool capable of automating the analysis of programs with variable reuse, successfully generating backward error bounds for several numerical programs previously beyond the reach of existing methods, while providing formal correctness guarantees.

This work addresses the inefficiency of joint program-and-parameter search in neuro-symbolic learning, where each candidate program requires separate parameter optimization. To overcome this bottleneck, the authors propose the Neural Differentiable Virtual Machine (NDVM), which applies automatic differentiation to the interpreter rather than individual programs. By decoupling symbolic structure from differentiable numeric state, NDVM preserves program dynamics while enabling precise backpropagation through execution traces. The design incorporates dense batched numeric buffers and runtime symbolic environment management, substantially amortizing evaluation overhead. Experiments demonstrate that NDVM achieves approximately 60× amortized speedup per-channel batching over baseline methods, exhibits near-linear multi-core scaling, and accelerates the discovery of high-quality solutions by roughly 24× under a fixed computational budget.