ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour

2026-07-26

AI Summary

This work addresses the high computational cost of Transformer inference under fully homomorphic encryption (FHE), which stems from the need to approximate nonlinear operations with polynomials. Existing approaches employ uniform approximation strategies that ignore the varying error tolerances across different layers, leading to suboptimal efficiency. To overcome this limitation, we propose ATLAS, the first framework that automates layer-wise adaptive polynomial approximation for FHE-friendly Transformers. We formulate the configuration problem as a multi-objective optimization balancing latency and accuracy, and accelerate the search via a two-stage constrained relaxation combined with surrogate modeling. Compatible with the CKKS scheme, ATLAS efficiently navigates an extremely large design space to jointly optimize performance and precision, enabling FHE deployment of models such as BERT, ViT, and LLaMA3 within one hour while significantly reducing inference latency and preserving high accuracy—thus overcoming the scalability bottlenecks of manual or uniform configuration strategies.

Abstract

Fully homomorphic encryption (FHE) provides strong cryptographic guarantees for private inference, but deploying transformer models under FHE remains prohibitively expensive. A key bottleneck is that non-linear operations such as softmax, normalization, and activation must be replaced with polynomial approximations compatible with the CKKS scheme, and the multiplicative depth consumed by these approximations dominates inference cost. Recent frameworks have advanced approximation techniques, yet all rely on manually configured approximation hyperparameters (e.g., number of iterations, polynomial degree), applied uniformly across all layers. While convenient, this uniform-configuration approach is overly rigid: different layers can tolerate different levels of approximation error without degrading predictive accuracy, and uniform configurations cannot exploit this variability to reduce latency. Allowing each layer to adopt its own configuration, however, causes the search space to explode with model depth, reaching roughly $10^{84}$ configurations for BERT/ViT (12 layers) and $10^{225}$ for LLaMA3 (32 layers), rendering manual exploration practically impossible. We present ATLAS, an automated framework that configures per-layer approximation settings by formulating the problem as a multi-objective optimization over latency and predictive accuracy. The resulting problem is inherently difficult: 1) competing objectives over a large decision space (120 or 320 variables for BERT/ViT or LLaMA3); 2) expensive evaluation, as each configuration takes 70-1,000 seconds even in cleartext; and 3) sparse optimization signals, as 35-50% of candidate configurations yield numerically invalid solutions. ATLAS addresses these challenges through a two-stage optimization strategy that progressively relaxes layer-wise constraints, combined with surrogate models to accelerate evaluation.

Problem

• homomorphic inference
• transformer approximation
• polynomial approximation
• multi-objective optimization
• FHE

Innovation

• automated approximation
• homomorphic inference
• multi-objective optimization
• polynomial approximation
• transformer efficiency