C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs

2026-07-23

AI Summary

This work addresses the significant performance degradation of multimodal large language models under post-training quantization, which stems from outlier channels highly sensitive to quantization. The authors propose a unified channel-level quantization method that, for the first time, incorporates task-specific Fisher information into sensitivity assessment. By introducing a Fisher-weighted objective that jointly accounts for task loss perturbations and quantization error, the method guides channel scaling to preserve critical channels—without requiring auxiliary modules such as LoRA. Combining Hessian approximation, channel-wise scaling, and post-training quantization, the approach achieves state-of-the-art results across eight benchmarks on Qwen2.5-VL, InternVL2, and LLaVA-OV, under both weight-only and weight-activation quantization settings.

Abstract

Multimodal large language models (MLLMs) require huge memory and computational costs, which limits their practical deployment. Post-training quantization (PTQ) techniques offer an efficient solution for model compression and inference acceleration. Yet, the quantized model faces performance degradation due to outlier channels, which are highly sensitive to quantization and substantially impair activation fidelity and task accuracy. To protect these salient channels during quantization, existing PTQ methods leverage modality- or token-level metrics to guide channel-wise scaling (CWS) of LLM decoders. However, these orthogonal measurements fail to capture channel-wise impacts on task-specific loss, and the misalignment between importance and scaling factors ultimately leads to suboptimal performance. To address this issue, we propose C-PTQ, a unified channel-wise PTQ method that harmonizes task-specific loss perturbation and quantization error. Motivated by second-order derivatives, we design a Fisher-weighted objective as a tractable Hessian approximation, seamlessly injecting task sensitivity into the scaling process. Notably, we achieve state-of-the-art performance without auxiliary modules like LoRA, thereby maintaining high efficiency. Experiments on Qwen2.5VL, InternVL2 and LLaVA-OV across 8 benchmarks demonstrate our effectiveness in both weight-only and weight-activation settings.

Problem

• post-training quantization
• multimodal large language models
• channel-wise sensitivity
• outlier channels
• task-specific loss

Innovation

• Post-training Quantization
• Channel-wise Sensitivity
• Fisher Information
• Multimodal LLMs
• Task-specific Loss