TALSC: Timeliness-Aware Large-Small VLM Collaboration for Infrastructure-Assisted Autonomous Driving

2026-08-03

AI Summary

This work addresses the challenge of deploying large vision-language models (VLMs) in autonomous driving systems, where onboard computational constraints limit model size, while roadside infrastructure—though capable of hosting large models—suffers from rapidly decaying information freshness under dynamic traffic conditions. To bridge this gap, the authors propose a timeliness-aware collaborative inference framework that jointly models the Age of Information (AoI) and VLM performance for the first time. They further design an online scheduling algorithm based on Lyapunov drift plus estimation penalty, achieving a theoretically guaranteed trade-off between communication latency and model accuracy. Experiments on the nuScenes dataset demonstrate that the proposed method improves Micro-F1 score by up to 12.6% and consistently outperforms baseline approaches across diverse communication and computational settings.

Abstract

The deployment of Vision-Language Models (VLMs) in autonomous driving (AD) systems is constrained by on-board computing power, restricting vehicles to small VLMs (SVLMs) with limited perception and reasoning capabilities. Infrastructure-assisted AD alleviates this resource constraint by enabling collaboration with large VLMs (LVLMs) at edge servers. However, in dynamic vehicular environments, the utility of sensory data for downstream tasks decays rapidly, making timeliness of information a critical concern. To balance the accuracy gains of LVLMs with their latency-induced timeliness degradation, we develop a Timeliness-Aware Large-Small VLM Collaboration (TALSC) framework. Specifically, we first model the Age of Information (AoI) evolution for VLM inference and characterize the coupling among AoI, token length, and task performance to formulate a general timeliness metric. Building on this, we propose the TALSC online scheduling algorithm. Since scheduling decisions have a delayed impact on future timeliness metric and the output token number is unknown at scheduling time, we design a Lyapunov drift-plus-estimated-penalty algorithm and provides a guaranteed performance. In simulation, we first conduct a case study to derive a fitted timeliness metric based on nuScenes dataset, and further show that TALSC outperforms baselines under various communication and computing settings, achieving up to a 12.6\% normalized improvement in Micro-F1 score compared with the best-performing baseline.

Problem

• Vision-Language Models
• Autonomous Driving
• Timeliness
• Age of Information
• Infrastructure-Assisted

Innovation

• Timeliness-Aware Scheduling
• Vision-Language Models
• Age of Information
• Infrastructure-Assisted Autonomous Driving
• Lyapunov Optimization