Trajectory Prediction Meets Large Language Models: A Survey

2025-06-03

AI Summary

This paper investigates how large language models (LLMs) can enhance semantic understanding and reasoning in trajectory prediction. Addressing the lack of unified modeling paradigms and interpretability in existing approaches, we propose the first language-trajectory fusion taxonomy, systematically categorizing five directions: language-based modeling for prediction, direct LLM-based prediction, language-guided scene understanding, language-driven data generation, and language-augmented reasoning. Methodologically, we integrate LLaMA/GPT-series models, multimodal encoders, prompt engineering, instruction tuning, chain-of-thought prompting, and language-trajectory alignment representation learning. Our contributions include: (1) a structured survey covering 120+ works; (2) distillation of key technical pathways—semantic modeling, cross-modal alignment, and causal reasoning; (3) definition of reproducible evaluation dimensions; and (4) advancement of paradigm shifts and standardization at the intersection of NLP and autonomous driving.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Recent advances in large language models (LLMs) have sparked growing interest in integrating language-driven techniques into trajectory prediction. By leveraging their semantic and reasoning capabilities, LLMs are reshaping how autonomous systems perceive, model, and predict trajectories. This survey provides a comprehensive overview of this emerging field, categorizing recent work into five directions: (1) Trajectory prediction via language modeling paradigms, (2) Direct trajectory prediction with pretrained language models, (3) Language-guided scene understanding for trajectory prediction, (4) Language-driven data generation for trajectory prediction, (5) Language-based reasoning and interpretability for trajectory prediction. For each, we analyze representative methods, highlight core design choices, and identify open challenges. This survey bridges natural language processing and trajectory prediction, offering a unified perspective on how language can enrich trajectory prediction.

Problem

• Integrating LLMs into trajectory prediction for autonomous systems
• Exploring language-driven techniques to enhance trajectory modeling
• Bridging NLP and trajectory prediction via unified methodologies

Innovation

• LLMs enhance trajectory semantic understanding
• Language-guided scene improves prediction accuracy
• Language-driven data generation for diverse scenarios