Video2LoRA: Parametric Video Internalization for Vision-Language Models

2026-06-02

AI Summary

This work addresses the high computational cost of existing vision-language models when processing videos, where inference scales linearly with frame count due to excessive visual token usage and redundant queries. The authors propose Video2LoRA, the first method to employ a Perceiver-based hypernetwork that generates low-rank adapters (LoRAs) in a single pass from the frozen intermediate representations of a vision-language model, thereby encoding video information parametrically. This enables question answering without any visual tokens during the query phase. The approach further supports composition of non-overlapping video segment LoRAs in rank space, offering a novel pathway for internalizing long videos. Experiments show that Video2LoRA matches the performance of direct video inference on five captioning benchmarks and most video question-answering tasks, stably handles inputs up to 1024 frames at 1024px resolution, reduces visual token load by up to 1500×, and decreases first-token latency by 6–80×.

Abstract

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Video2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Video2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Video2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Video2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.

Problem

• video processing
• vision-language models
• computational cost
• token efficiency
• video internalization

Innovation

• Video2LoRA
• Low-Rank Adaptation
• Vision-Language Models
• Parametric Internalization
• Token-Efficient Inference