Adding complex numbers to expression template algorithmic differentiation tools

2025-08-07

AI Summary

Existing expression-template-based operator-overloading automatic differentiation (AD) tools (e.g., CoDiPack) lack native complex-number support, forcing complex operations to be decomposed into redundant real-valued arithmetic, inflating computational graph size and derivative tape memory overhead. Method: We propose a comprehensive complex-number integration scheme within the expression-template AD framework, encapsulating complex arithmetic and elementary functions—including optimized complex linear algebra—directly in the template logic, thereby avoiding explicit real/imaginary part splitting. Contribution/Results: This work presents the first end-to-end complex AD implementation in an expression-template AD system. Experiments on synthetic benchmarks show up to 42% reduction in memory footprint and up to 3.1× speedup in gradient computation, significantly improving differentiation efficiency and scalability for complex-valued applications.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Operator overloading algorithmic differentiation (AD) tools are usually only developed for floating-point values. Algorithmic optimization for, e.g., linear systems solvers or matrix-matrix multiplications are often introduced via external functions or manual function specializations. Complex numbers can be viewed as aggregates of two floating-point values on which specialized operations are applied. Typically, these operations can be handled by the regular floating-point operations from the AD tool. Nevertheless, adding the complex number operations to the expression template framework of modern operator overloading AD tools has several benefits. The internal computations of a complex number operation are hidden, and the complex operations do not decompose into single operations. This leads to a smaller memory footprint of the recorded tape and faster gradient computation times. We will discuss these problems, analyze how complex numbers can be integrated into modern operator overloading AD tools, demonstrate an implementation in CoDiPack, and show performance results on a synthetic test case.

Problem

• Extend AD tools to support complex number operations
• Optimize memory and speed in gradient computations
• Integrate complex numbers into expression template frameworks

Innovation

• Extends AD tools with complex number operations
• Hides internal computations in expression templates
• Reduces memory usage and speeds gradients