V-LynX: Token Interface Alignment for Video+X LLMs

2026-05-29

AI Summary

This work addresses the challenge of efficiently integrating novel modalities—such as audio, 3D, and multi-view data—into existing video large language models without relying on modality-specific encoders or paired supervision. The study reveals, for the first time, a manipulable continuous manifold of visual token interfaces within video language models and introduces V-LynX, a framework that employs a lightweight auxiliary pathway operating in parallel with a frozen visual encoder. By leveraging only unpaired, single-modality data, V-LynX aligns the attention responses and statistical distributions of new modalities with those of the model’s video priors. This approach preserves the original architecture while enabling flexible extension to diverse perceptual modalities, achieving state-of-the-art performance on tasks including audio-visual question answering, 3D reasoning, and multi-view video understanding, all with high computational efficiency.

Abstract

This study introduces an intriguing phenomenon in Video LLMs: rather than merely translating frames into textual embeddings, Video LLMs establish a continuous manifold, token interface, allowing visual tokens to operate as standalone entities within the architecture. Exploiting this discovery, we propose V-LynX, a scalable framework that integrates novel modalities into Video LLMs by repurposing the internalized interface. Departing from conventional paradigms that necessitate heavy modality-specific encoders or paired supervision, V-LynX employs a lightweight auxiliary pathway in parallel with the frozen vision encoder. Our method integrates new sensory inputs with intrinsic video priors by aligning both attention responses and statistical distributions using unpaired unimodal data sets. This ensures manifold compatibility while preserving the integrity of the Video LLMs. Extensive benchmarks demonstrate that V-LynX achieves SOTA and efficiency across audio-visual QA, 3D reasoning, high-frame-rate, and multi-view video understanding. The code is available at https://github.com/park-jungin/lynx.

Problem

• Video LLMs
• modality integration
• token interface
• unpaired data
• multimodal alignment

Innovation

• token interface
• Video LLMs
• modality alignment
• unpaired learning
• manifold compatibility