A Comprehensive Study of Structural Pruning for Vision Models

2024-06-18

AI Summary

Existing structured pruning research lacks a unified evaluation protocol, hindering fair and systematic comparison across methods. Method: We introduce PruningBench—the first standardized benchmark for structured pruning—featuring a three-dimensional, consistent evaluation framework spanning models (CNNs and Vision Transformers), tasks (image classification and object detection), and metrics. It systematically evaluates 16 state-of-the-art pruning methods under controlled, reproducible conditions. Contribution/Results: PruningBench establishes the first cross-architecture, cross-task standardized evaluation paradigm for structured pruning, enabling customizable pruning experiments via modular, open-source APIs. It provides a fully reproducible experimental framework, an online interactive visualization platform, and a real-time leaderboard. All code, configuration files, and benchmark results are publicly released; the platform is live and actively maintained.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Structured Learning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Structural pruning has emerged as a promising approach for producing more efficient models. Nevertheless, the community suffers from a lack of standardized benchmarks and metrics, leaving the progress in this area not fully comprehended.To fill this gap, we present the first comprehensive benchmark, termed PruningBench, for structural pruning. PruningBench showcases the following three characteristics: 1) PruningBench employs a unified and consistent framework for evaluating the effectiveness of diverse structural pruning techniques; 2) PruningBench systematically evaluates 16 existing pruning methods, encompassing a wide array of models (e.g., CNNs and ViTs) and tasks (e.g., classification and detection); 3) PruningBench provides easily implementable interfaces to facilitate the implementation of future pruning methods, and enables the subsequent researchers to incorporate their work into our leaderboards. We provide an online pruning platform http://pruning.vipazoo.cn for customizing pruning tasks and reproducing all results in this paper. Leaderboard results can be available on https://github.com/HollyLee2000/PruningBench.

Problem

• Standardization
• Evaluation Methodology
• Structural Pruning Techniques

Innovation

• PruningBench
• Unified Evaluation
• Comprehensive Benchmark