Algorithmic differentiation for domain specific languages in C++ with expression templates

2026-02-17

AI Summary

This work addresses the inefficiencies of conventional C++ automatic differentiation (AD) approaches based on operator overloading, which disrupt original memory layouts, impede compiler optimizations such as SIMD vectorization, and fail to recognize domain-specific operations like linear algebra primitives. To overcome these limitations, the authors propose a domain-specific AD framework for C++ that leverages expression templates to bind identifiers to high-level semantic entities—such as matrices—and employs operation annotation and code generation to enable customized reverse-mode AD. By preserving native memory layouts, the approach facilitates compiler optimizations while applying efficient, domain-aware differentiation strategies. Evaluated on pure linear algebra programs, the method incurs a runtime overhead below 4.5× the theoretical lower bound, achieving performance comparable to state-of-the-art specialized AD tools.

Technology Category

• Application Domains: Software Engineering
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The application of operator overloading algorithmic differentiation (AD) to computer programs in order to compute the derivative is quite common. But, the replacement of the underlying computational floating point type with the specialized type of an AD tool has two problems. First, the memory structure of the program is changed and floating-point data is interleaved with identifiers from AD. This prevents the compiler from performing optimizations such as SIMD optimizations. Second, the AD tool does not see any domain-specific operations, e.,g. linear algebra operations, that the program uses. This prevents the AD tool from using specialized algorithms in such places. We propose a new AD tool that is tailored to such situations. The memory structure of the primal data is retained by associating an identifier with each entity, e.,g. matrix, and not with each floating point value, e.,g. element of the matrix. Operations on such entities can then be annotated and a generator is used to create the AD overloads. We demonstrate that this approach provides performance comparable to that of other specializations. In addition, the run-time factor is below the theoretical 4.5 of reverse AD for programs that are written purely with linear algebra entities and operations.

Problem

• algorithmic differentiation
• domain-specific languages
• expression templates
• memory layout
• linear algebra operations

Innovation

• algorithmic differentiation
• expression templates
• domain-specific languages
• linear algebra
• reverse mode AD