FROST-Drive: Scalable and Efficient End-to-End Driving with a Frozen Vision Encoder

2026-01-06

arXiv.org

AI Summary

This work addresses the limited generalization of end-to-end autonomous driving models in novel and complex scenarios, where full fine-tuning of vision encoders often leads to overfitting. To mitigate this, the authors propose freezing the vision encoder of a pretrained vision-language model (VLM) to preserve generic visual knowledge, while introducing a Transformer-based multimodal adapter to effectively fuse visual and language features. A GRU decoder is employed to generate smooth driving trajectories. Furthermore, a custom loss function tailored to the Rater Feedback Score (RFS) is designed to optimize driving behavior quality. Evaluated on the Waymo Open E2E dataset under long-tail scenarios, the proposed approach significantly outperforms full fine-tuning baselines, demonstrating that freezing the VLM encoder enhances model robustness and generalization.

Technology Category

Computer Vision: Large Vision ModelsMachine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

End-to-end (E2E) models in autonomous driving aim to directly map sensor inputs to control commands, but their ability to generalize to novel and complex scenarios remains a key challenge. The common practice of fully fine-tuning the vision encoder on driving datasets potentially limits its generalization by causing the model to specialize too heavily in the training data. This work challenges the necessity of this training paradigm. We propose FROST-Drive, a novel E2E architecture designed to preserve and leverage the powerful generalization capabilities of a pretrained vision encoder from a Vision-Language Model (VLM). By keeping the encoder's weights frozen, our approach directly transfers the rich, generalized world knowledge from the VLM to the driving task. Our model architecture combines this frozen encoder with a transformer-based adapter for multimodal fusion and a GRU-based decoder for smooth waypoint generation. Furthermore, we introduce a custom loss function designed to directly optimize for Rater Feedback Score (RFS), a metric that prioritizes robust trajectory planning. We conduct extensive experiments on Waymo Open E2E Dataset, a large-scale datasets deliberately curated to capture the long-tail scenarios, demonstrating that our frozen-encoder approach significantly outperforms models that employ full fine-tuning. Our results provide substantial evidence that preserving the broad knowledge of a capable VLM is a more effective strategy for achieving robust, generalizable driving performance than intensive domain-specific adaptation. This offers a new pathway for developing vision-based models that can better handle the complexities of real-world application domains.

Problem

• end-to-end driving
• generalization
• vision encoder
• autonomous driving
• long-tail scenarios

Innovation

• frozen vision encoder
• vision-language model
• end-to-end driving
• generalization
• Rater Feedback Score