Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

2026-08-07

AI Summary

This work addresses the challenge of achieving general-purpose, efficient compression of vision-language models without relying on task-specific data or costly retraining. The authors propose PORTA, a novel framework that enables universal pruning by estimating cross-modal activation variability using a generic calibration set, thereby constructing a task- and modality-agnostic importance metric. PORTA further incorporates an adaptive layer-wise sparsity allocation mechanism to optimize compression efficiency. Evaluated on prominent vision-language models—including CLIP, BLIP, and Qwen2-VL—PORTA consistently outperforms existing retraining-free pruning methods under high compression ratios while preserving strong downstream task performance.

Abstract

Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments. Existing pruning strategies often depend on task-specific criteria or LLM-oriented importance measures, making them unsuitable for task-agnostic pruning, where no task-specific samples are available at pruning time and the pruned model remains broadly applicable. We introduce a retraining-free VLM pruning framework called PORTA that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities. PORTA further incorporates an adaptive sparsity allocation mechanism that assigns layer-wise pruning ratios based on output feature variability, avoiding the limitations of uniform sparsity and reducing performance degradation at high compression levels. Extensive experiments across VLM architectures, such as CLIP, BLIP, and Qwen2-VL, demonstrate that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression. Code is available at https://github.com/cau-hai-lab/PORTA.git.

Problem

• vision-language models
• task-agnostic pruning
• model compression
• retraining-free
• sparsity

Innovation

• task-agnostic pruning
• retraining-free compression
• activation variation
• adaptive sparsity allocation
• vision-language models