Integrated electro-optic attention nonlinearities for transformers

2026-04-10

AI Summary

This work addresses the inference latency bottleneck in Transformers caused by nonlinear operations such as Softmax, despite their low computational cost. The authors propose, for the first time, an analog optoelectronic nonlinear computing unit based on thin-film lithium niobate (TFLN) Mach–Zehnder modulators to efficiently implement Softmax and Sigmoid functions. Operating with only 4-bit quantization, the approach maintains high accuracy, supports high-speed 10 GBaud encoding, and demonstrates strong robustness under noisy conditions. Experimental results show that the method achieves accuracy comparable to purely digital implementations in both vision Transformers and large language models while significantly reducing the latency of nonlinear computations, thereby validating the feasibility of TFLN-based devices for accelerating attention mechanisms in hybrid integrated hardware systems.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance

Abstract

Transformers have emerged as the dominant neural-network architecture, achieving state-of-the-art performance in language processing and computer vision. At the core of these models lies the attention mechanism, which requires a nonlinear, non-negative mapping using the Softmax function. However, although Softmax operations account for less than 1% of the total operation count, they can disproportionately bottleneck overall inference latency. Here, we use thin-film lithium niobate (TFLN) Mach-Zehnder modulators (MZMs) as analog nonlinear computational elements to drastically reduce the latency of nonlinear computations. We implement electro-optic alternatives to digital Softmax and Sigmoid, and evaluate their performance in Vision Transformers and Large Language Models. Our system maintains highly competitive accuracy, even under aggressive 4-bit input-output quantization of the analog units. We further characterize system noise at encoding speeds up to 10 GBaud and assess model robustness under various noise conditions. Our findings suggest that TFLN modulators can serve as nonlinear function units within hybrid co-packaged hardware, enabling high-speed and energy-efficient nonlinear computation.

Problem

• attention mechanism
• Softmax bottleneck
• nonlinear computation
• inference latency
• transformers

Innovation

• electro-optic nonlinearity
• thin-film lithium niobate
• Mach-Zehnder modulator
• analog Softmax
• hybrid co-packaged hardware