Keep the Needle, Prune the Haystack: Defect-Preserving Token Pruning for Efficient Zero-Shot Anomaly Detection

2026-08-04

AI Summary

This work addresses the inefficiency of existing zero-shot anomaly detection methods, which suffer from computationally intensive operations and poor sensitivity to subtle anomalies due to conventional token pruning. To overcome these limitations, the authors propose KeepAD, a novel framework that ensures high recall in shallow layers through local 2×2 neighborhood coverage and a deterministic anomaly retention mechanism. In deeper layers, it introduces frozen normal/anomaly prototypes to guide adaptive token pruning and employs a dense-to-sparse self-distillation strategy to refine early token routing. KeepAD is the first to integrate high-recall, anomaly-aware asymmetric token pruning into the zero-shot setting, achieving a token retention rate below 20% across 13 industrial and medical benchmarks while incurring average image- and pixel-level AUROC drops of no more than 2.7 percentage points and delivering up to 7.9× inference speedup.

Abstract

Zero-shot visual anomaly detection has achieved remarkable progress, with recent vision-only approaches further improving performance while simplifying the inference pipeline. However, existing methods typically perform dense computation over all images and spatial tokens, despite the fact that normal samples dominate real-world scenarios and anomalies usually occupy only small regions. Token pruning offers a promising solution, but introduces an asymmetric pruning risk in anomaly detection: retaining normal tokens mainly incurs redundant computation, whereas removing anomalous tokens may eliminate the only evidence for detection and localization. This risk is particularly severe in early layers, where pruning provides the greatest computational benefit but anomaly semantics remain unreliable. We propose KeepAD, a defect-preserving token pruning framework that formulates token selection as high-recall, anomaly-aware routing. In shallow layers, KeepAD combines coverage-preserving selection over local $2\times2$ patch neighborhoods with deterministic anomaly rescue to reduce the risk of discarding subtle defects. In deeper layers, frozen normal and abnormal prototypes guide pruning under an image-adaptive token budget, aggressively removing low-risk normal tokens while preserving local anomaly evidence. Dense-to-sparse self-distillation further supervises early token routing without introducing additional inference overhead. Experiments on six industrial and seven medical zero-shot anomaly detection benchmarks show that KeepAD reduces the token retention ratio to below $20\%$, while limiting the average degradation in image-level and pixel-level AUROC to within $2.7$ percentage points. At the most aggressive operating point, KeepAD achieves a $7.9\times$ speedup over the strongest CLIP-based baseline.

Problem

• zero-shot anomaly detection
• token pruning
• defect preservation
• asymmetric pruning risk
• efficient inference

Innovation

• token pruning
• zero-shot anomaly detection
• defect preservation
• anomaly-aware routing
• self-distillation