Enabling Fully Integer-Only Inference for Lightweight Detection Transformers

2026-07-27

AI Summary

This work addresses the challenge of deploying lightweight Detection Transformers (DETRs) on neural processing units (NPUs) and microcontrollers, which is hindered by non-integer operations such as Softmax, GELU, and LayerNorm. To overcome this, the authors propose I-LW-DETR, the first end-to-end fully integer-aware lightweight DETR that quantizes all components—including deformable attention, feature fusion, and nonlinear activations—into integer-only operations. Key innovations include scale-preserving grouped convolution, sign-dependent ShiftGELU, and constrained Shiftmax normalization, which jointly enable hardware-friendly design without compromising accuracy. Experimental results demonstrate that I-LW-DETR incurs only marginal accuracy degradation across various model scales while achieving a 3.6× reduction in model size and over an order-of-magnitude decrease in computational cost.

Abstract

Vision Transformer detectors now approach the accuracy of CNNs but remain difficult to deploy on NPUs and microcontrollers because key components, including deformable attention, feature fusion, and nonlinear activation functions, are not natively compatible with integer arithmetic. Existing quantized detectors either retain operators such as Softmax, GELU, and LayerNorm or focus on heavyweight backbones, leaving lightweight detection transformers without an end-to-end integer implementation. We address this gap with I-LW-DETR, the first fully integer-only lightweight DETR, in which every operation in the forward pass, including transformer nonlinearities, is executed in integer arithmetic. I-LW-DETR is built upon three key components: a scale-preserving split convolution that assigns independent activation scale to each branch of the multi-scale projector; SD-ShiftGELU, a sign-dependent GELU approximation that preserves element-wise behavior while avoiding the accuracy degradation; and a constrained Shiftmax that maintains stable Softmax normalization. Experimental results demonstrate that the proposed quantization pipeline consistently produces efficient fully integer-only models across different model scales. Across all model scales, the proposed pipeline incurs only a moderate accuracy degradation while reducing the model size by approximately $3.6\times$ and the computational cost by more than one order of magnitude.

Problem

• integer-only inference
• lightweight detection transformers
• quantization
• NPU deployment
• nonlinear activation functions

Innovation

• integer-only inference
• lightweight DETR
• quantization
• ShiftGELU
• Shiftmax