PRISM: Sensitivity-Aware PolynoMial PRuning for EffIcient Neural Network Encryption

2026-07-20

AI Summary

This work addresses the critical yet underexplored impact of structured pruning on the reliability of homomorphic encryption-based neural network inference. We propose Polynomial Sensitivity-Aware Pruning (PSAP), the first structured pruning framework that explicitly incorporates reliability awareness by jointly evaluating weight magnitudes, polynomial activation sensitivity, and rotation overhead to guide pruning toward fault-tolerant regions. PSAP identifies fault-critical layers comprising only 1.1% of model parameters and enables efficient selective hardening. Experiments across diverse architectures and datasets demonstrate that PSAP reduces the number of layers suffering catastrophic accuracy degradation from 5–14 to at most 2, achieving up to a 29× reduction in worst-case vulnerability. Additionally, it decreases rotation operations by 45.2% and lowers multiplicative depth from 66 to 56, thereby enabling bootstrapping-free hierarchical inference.

Abstract

Structured pruning is essential for making neural network inference feasible under homomorphic encryption (HE), yet its impact on model reliability has remained unexplored. This paper presents a systematic reliability characterization of pruned CKKS-encrypted neural networks and introduces Polynomial-Sensitivity-Aware Pruning (PSAP), a structured pruning method that is inherently reliability-aware. PSAP scores filters jointly by weight magnitude, polynomial activation sensitivity, and rotation cost, which concentrates pruning in fault-tolerant regions. Across two architectures, two datasets, two numerical representations, and five bit-error rates (40 full-model and 108 per-layer experiments), PSAP-pruned models limit catastrophic (>10 pp accuracy drop) layers to at most two versus 5--14 for magnitude-pruned baselines, reducing worst-case vulnerability by up to 29 times under int32 bit-flip injection. Direct CKKS encrypted fault injection indicates a safe operating boundary near BER~ 10^{-5}, supporting int32 injection as a conservative reliability proxy. The fault-critical structural layers account for only 1.1% of parameters, enabling selective hardening at minimal overhead. These reliability gains are obtained alongside competitive efficiency: PSAP reduces Halevi--Shoup rotations by up to 45.2\% on ResNet-32, and an adaptive mixed-degree allocation scheme lowers multiplicative depth from 66 to 56 levels, enabling leveled inference without bootstrapping.

Problem

• structured pruning
• homomorphic encryption
• model reliability
• CKKS
• fault tolerance

Innovation

• structured pruning
• homomorphic encryption
• reliability-aware
• polynomial sensitivity
• CKKS