Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

2026-06-08

AI Summary

This work addresses the challenge of deploying large language models on edge devices, where latency, memory, and energy constraints severely limit practicality, and existing pruning methods struggle to balance performance and efficiency. The authors propose a hardware-aware, multi-objective structured pruning framework that first removes entire attention and MLP blocks at a coarse granularity and then employs parallel Bayesian optimization to jointly determine layer-wise pruning ratios while incorporating importance scores for fine-grained removal of attention heads, MLP dimensions, and network depth. By integrating multi-objective optimization with explicit hardware constraints, the approach avoids local optima and enhances deployment stability. Experiments demonstrate that at 37.5% and 50% pruning rates, the method significantly reduces inference overhead while outperforming state-of-the-art techniques on commonsense reasoning and zero-shot tasks, achieving a superior trade-off among accuracy, latency, and model size.

Abstract

Large Language Models (LLMs) have achieved widespread adoption because of their strong reasoning and query-response capabilities. However, deploying them in embedded and edge computing environments remains challenging because of strict latency, memory, and energy constraints. Their large parameter counts and computational demands hinder efficient execution on resource-constrained platforms. Although model pruning has emerged as a viable solution for reducing scale while preserving performance, jointly optimizing layers, attention heads, and Multi-Layer Perceptron (MLP) dimensions remains highly complex. Exhaustively exploring this combined design space is computationally expensive and often leads to local optima or unstable configurations. To address these limitations, we propose a hardware-aware, multi-objective structured pruning framework. The proposed two-stage method explicitly targets latency and model size for efficient deployment on edge devices. In the coarse-grained stage, multi-objective depth pruning removes entire attention and MLP blocks to reduce computational load and memory usage. In the subsequent fine-grained stage, Parallel Bayesian Optimization (PBO) searches for the optimal layer-wise pruning ratios for pruning under latency constraints, while importance-based strategies rank the specific components to be pruned within each layer's allocated budget. Experimental results show that our approach reduces model complexity with minimal impact on commonsense reasoning tasks and zero-shot performance. Our method achieves a favorable trade-off among accuracy, latency, and model size, making it suitable for edge deployment. Across multiple LLMs at 37.5% and 50% pruning ratios, the proposed approach achieves better performance on commonsense reasoning tasks than existing methods while significantly reducing inference cost.

Problem

• Large Language Models
• Model Pruning
• Latency Optimization
• Model Size Reduction
• Edge Deployment

Innovation

• structured pruning
• multi-objective optimization
• Bayesian optimization
• hardware-aware
• edge deployment