OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

2026-08-04

AI Summary

This work addresses the high computational cost incurred by multimodal large language models when processing lengthy audio-visual token sequences, a challenge exacerbated by existing compression methods that suffer significant performance degradation under low token budgets. To overcome this, the authors propose OmniPack, a training-free, unified token compression framework that jointly optimizes structured pre-LLM compression and task-aware semantic refinement within the LLM. Its key innovations include modality-specific importance scoring, a global coverage strategy, similarity-aware token merging, and mechanisms for text-guided and cross-modal coordination. Extensive experiments demonstrate that OmniPack achieves state-of-the-art performance-efficiency trade-offs across five benchmarks and three Omni-LLM backbones, retaining 92.9% of the original model’s performance on Qwen2.5-Omni-7B while consuming only 6.8% of the original FLOPs.

Abstract

Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.

Problem

• token compression
• omni-modal LLMs
• computational overhead
• audio-visual redundancy
• efficient deployment

Innovation

• token compression
• omni-modal LLM
• training-free framework
• audio-visual collaboration
• structural redundancy