Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models

2026-07-25

AI Summary

This work addresses the high inference prefill latency and GPU memory overhead of omnimodal large language models when processing synchronized audio-visual inputs, a challenge exacerbated by long input sequences and exacerbated further by existing pruning methods that neglect cross-modal dependencies and user query semantics. The paper proposes the first training-free, query-aware audio-visual joint pruning framework, which integrates adaptive temporal windowing, audio saliency, cross-modal attention, and text-query relevance into a unified scoring mechanism. By pairing audio-visual tokens and applying K-medoids representative sampling, the method efficiently compresses inputs while preserving task-critical cross-modal information. Experiments demonstrate that the approach achieves up to 3.25× prefill speedup and 1.3× reduction in memory usage while retaining over 99% of the original model performance, significantly outperforming current baselines.

Abstract

Omnimodal large language models (OmniLLMs) are rapidly extending multimodal reasoning to cover synchronized audio and video. However, the resulting audio-video token sequences are long, leading to high prefill latency and GPU memory usage at inference time. Existing token pruning methods, designed mainly for vision-only inputs, miss both the cross-modal links between audio and video and the user query that decides which content matters. To bridge this gap, we present Omni-Prune, a training-free, query-aware audio-visual token pruning framework that jointly removes redundancy from both modalities while keeping task-relevant cross-modal evidence. Specifically, Omni-Prune first splits the token sequence into adaptive time windows placed at audio saliency peaks, then scores audio and video tokens on a single scale that combines encoder attention with text-query relevance, and pairs related audio-video tokens so that they are kept together. Within each window, a final K-medoids step then selects a few representative tokens, adding diverse cues that score-based selection alone would miss. Extensive experiments demonstrate that Omni-Prune outperforms established baseline methods, delivering up to 3.25x prefill speedup and 1.3x memory reduction while retaining over 99% of full-model performance.

Problem

• omnimodal
• token pruning
• audio-video
• query-aware
• large language models

Innovation

• token pruning
• omnimodal LLMs
• query-aware
• cross-modal alignment
• training-free