OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

2026-07-21

AI Summary

This work addresses the challenges of long-form audiovisual reasoning, where critical evidence is sparse and distributed across modalities, and high-fidelity inputs incur prohibitive computational costs that existing full-modality large language models struggle to handle efficiently. The authors propose a tool-augmented post-training framework in which the model first constructs a low-cost global preview and then selectively invokes localized high-fidelity zoom-in tools for fine-grained analysis as needed. A novel TimeAnchor mechanism ensures temporal consistency across multiple granularities, while a temporally enhanced data engine automatically generates tool-use trajectories without manual annotation. Through joint optimization via supervised fine-tuning and reinforcement learning, the method significantly improves answer accuracy and temporal localization on multiple audiovisual benchmarks, while concentrating high-fidelity computation on information-dense regions.

Abstract

Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call -- a sparse global preview and a denser local clip -- we introduce TimeAnchor, which keeps the tool's temporal argument valid and round-trip-consistent across these granularities, rather than tied to frame indices from a particular sampling rate. To make this tool-use behavior trainable without expensive manual interval annotation, we build a Temporal Augmented Data Engine that synthesizes tool-use post-training trajectories by video editing and composition. Experiments across omnimodal and video benchmarks show that OmniReasoner improves both answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code is available at https://github.com/RockyChen0205/OmniReasoner.

Problem

• long audio-video reasoning
• omnimodal LLMs
• sparse evidence
• cross-modal
• high-fidelity inputs

Innovation

• tool use
• long audio-video reasoning
• temporal grounding
• TimeAnchor
• omnimodal LLM