Thermodynamic Bound on Energy and Negentropy Costs of Inference in Deep Neural Networks

2025-03-13

AI Summary

Deep neural network (DNN) inference incurs irreversible energy dissipation primarily due to nonlinear activation functions, yet fundamental thermodynamic limits on energy and negentropy consumption remain unestablished. Method: This work systematically applies Landauer’s principle to end-to-end DNN inference modeling, distinguishing reversible linear operations from intrinsically irreversible activations; it replaces energy with negentropy as a universal metric for information processing cost and integrates reversible computing theory, information-theoretic entropy analysis, and DNN dynamical modeling. Contribution/Results: We derive rigorous lower bounds on both energy and negentropy consumption per inference step—expressed as a function of the number of neurons undergoing state transitions. These bounds provide the first physically grounded constraints for reversible analog AI hardware and establish principled, energy-efficient architectural design guidelines for next-generation neuromorphic systems.

Technology Category

Machine Learning: Hardware-aware MLKnowledge Representation and Reasoning: Computational Complexity of ReasoningCognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applicationsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The fundamental thermodynamic bound is derived for the energy cost of inference in Deep Neural Networks (DNNs). By applying Landauer's principle, we demonstrate that the linear operations in DNNs can, in principle, be performed reversibly, whereas the non-linear activation functions impose an unavoidable energy cost. The resulting theoretical lower bound on the inference energy is determined by the average number of neurons undergoing state transition for each inference. We also restate the thermodynamic bound in terms of negentropy, a metric which is more universal than energy for assessing thermodynamic cost of information processing. Concept of negentropy is further elaborated in the context of information processing in biological and engineered system as well as human intelligence. Our analysis provides insight into the physical limits of DNN efficiency and suggests potential directions for developing energy-efficient AI architectures that leverage reversible analog computing.

Problem

• Derives thermodynamic bound on energy cost for DNN inference.
• Identifies unavoidable energy cost from non-linear activation functions.
• Explores negentropy as universal metric for thermodynamic cost in AI.

Innovation

• Applies Landauer's principle to DNN energy costs
• Defines thermodynamic bound using neuron state transitions
• Introduces negentropy for universal thermodynamic cost assessment