APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning

2026-08-05

AI Summary

This work addresses the high computational cost and inference latency of deep neural networks on edge devices, exacerbated by the limitations of existing compression methods that rely on manual hyperparameter tuning, lack cross-architecture generalizability, and apply uniform strategies that ignore inter-layer sensitivity differences, often leading to significant accuracy degradation. To overcome these challenges, we propose APQF, a novel framework that, for the first time, integrates a large language model-based planner with an empirical profiling agent to enable fully automated, training-aware joint pruning and quantization for both CNNs and Vision Transformers. By analyzing layer-wise performance and sensitivity, APQF adaptively determines structured pruning ratios, mixed-precision bitwidths, and fine-tuning strategies, validating each configuration prior to execution. On ImageNet-1k, our method reduces computation to 5.6–7.7% of the original (13–18× compression) while achieving state-of-the-art Top-1 accuracy; notably, VGG7 attains 93.15% accuracy at merely 0.41% of its original computational cost, even surpassing its full-precision baseline.

Abstract

Modern deep neural networks achieve strong performance, but their scale makes them costly and slow, especially on resource-constrained edge devices. Pruning and quantization address this, but rely on manual, expert choices and on algorithms that are hard to apply across architectures. Uniform settings also ignore how differently individual layers respond to compression, which costs accuracy. We introduce APQF, an agentic profiling-guided framework that combines structured pruning, mixed-precision quantization-aware training, and accuracy recovery in one automated pipeline. A profiling agent measures how cost is distributed across the model and how sensitive each part is to pruning, and this evidence drives per-layer pruning ratios, per-layer bit-widths, and the recovery strategy, all proposed by LLM planners and validated before execution. To our knowledge, APQF is the first framework to combine LLM-guided, profiling-grounded decisions with a fully training-aware pruning and quantization pipeline for both CNNs and vision transformers. We evaluate APQF on ResNet, VGG7, ViT, DeiT, and Swin using ImageNet-1k and CIFAR-10. On ImageNet it cuts compute to 5.6-7.7 percent of the original bit-operations, a 13-18x reduction, while keeping accuracy close to the baseline, and under a 200K-image budget it stays roughly 17 points higher in Top-1 than existing joint pruning and quantization methods. On CIFAR-10 it compresses further than that method on four of five architectures. On VGG7 it reaches 93.15 percent using only 0.41 percent of baseline bit-operations, the only method at that compression level to improve on its full-precision baseline. Ablations show that uniform compression loses the most accuracy at matched compute, and that withholding profiling data from the planner hurts every model. Six LLM planners, including free open-weight ones, all reach 97.4-97.9 percent on Swin-Tiny.

Problem

• model compression
• structured pruning
• mixed-precision quantization
• edge devices
• accuracy loss

Innovation

• Agentic Profiling
• LLM-guided Compression
• Structured Pruning
• Mixed-Precision Quantization
• Adaptive Fine-Tuning