Kernel-Based ReLU Approximation for Homomorphic Encryption-Compatible Privacy-preserving Deep Learning Models

2026-05-22

AI Summary

Homomorphic encryption supports only addition and multiplication, making it challenging to efficiently implement nonlinear activation functions commonly used in deep learning—such as ReLU—and thereby hindering the deployment of privacy-preserving large language models. This work proposes a kernel-based approach inspired by Jackson’s theorem to construct a quadratic polynomial smooth approximation of ReLU with low multiplicative depth, achieving high fidelity while remaining compatible with homomorphic encryption schemes. To the best of our knowledge, this is the first study to introduce a kernel-based smooth ReLU into the encrypted domain, with training and evaluation conducted on token embeddings of pretrained Transformer models. Experiments demonstrate that the proposed method achieves both high approximation accuracy and practical utility across multiple NLP tasks, significantly advancing the feasibility of privacy-preserving large language models under encrypted inference.

Abstract

As privacy concerns in AI technologies continue to grow, Homomorphic Encryption (HE) offers a way to perform computations on encrypted data without the need of decryption during operations. However, HE is limited to addition and multiplication, making non-linear functions incompatible in their original form. This limitation has become more critical with the widespread use of Large Language Models (LLMs), where the non-linearity of activation functions such as the Rectified Linear Unit (ReLU) poses challenges for deployment in privacy-preserving Natural Language Processing (NLP) settings. This paper proposes a kernel-based approximation of ReLU, enabling its use within HE-constrained settings and thus contributing a critical step toward supporting privacy-preserving LLMs. A smooth kernel-based function, mimicking ReLU, is approximated using a second-degree polynomial, inspired by Jackson's theorem, to achieve low multiplicative depth. The proposed method is trained and assessed directly on token embeddings from pre-trained LLMs and evaluated in various scenarios, from simulated and tokenized data to deep learning and transformer models. Results show improved approximation fidelity, supporting the method's suitability for secure and privacy-preserving inference in various tasks.

Problem

• Homomorphic Encryption
• ReLU
• Privacy-preserving Deep Learning
• Large Language Models
• Non-linear Activation Functions

Innovation

• Homomorphic Encryption
• ReLU Approximation
• Kernel-based Function
• Multiplicative Depth
• Privacy-preserving LLMs