TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents

2026-05-31

AI Summary

Existing travel planning evaluation benchmarks overemphasize constraint satisfaction while neglecting multidimensional quality aspects such as spatiotemporal costs, and suffer from insufficient data authenticity and coverage, limiting their ability to comprehensively assess LLM-driven planning capabilities. This work proposes the first six-dimensional evaluation framework encompassing accuracy, compliance, temporal feasibility, spatial coherence, economic efficiency, and practicality. It introduces a high-fidelity simulation sandbox grounded in real-world transportation and accommodation data, integrating geospatial APIs, fine-grained queuing models, and multi-agent simulation to enable dynamic, end-to-end assessment of travel itineraries. Experiments across twelve state-of-the-art methods reveal that current LLMs exhibit significant deficiencies in spatiotemporal reasoning and budget management, and that sophisticated agent-based strategies do not consistently yield performance improvements.

Abstract

The development of Large Language Models (LLMs) has significantly improved travel planning applications, yet evaluating such models is limited by existing benchmarks' limitations: 1) overemphasis on constraint compliance, neglecting multi-dimensional qualities like spatio-temporal cost; 2) datasets lacking real-world authenticity and coverage in key areas (e.g., lodging, transport); and 3) isolated daily plan assessments that miss critical details (e.g., the impact of daily accommodation and visit pacing) needed for entire plan's evaluation. To address this gap, we introduce TravelEval, a realistic and comprehensive benchmark. TravelEval features 1) a novel six-dimensional evaluation framework to holistically assess plans across accuracy, compliance, temporality, spatiality, economy, and utility dimensions; 2) a highly realistic data sandbox with precise accommodation pricing and authentic intercity transportation data; and 3) a simulation-based global evaluation method that emulates complete travel plans with API-integrated geographic information and fine-grained queuing time. Evaluating 12 mainstream approaches with TravelEval reveals several valuable insights, such that LLMs struggle with globally-optimized multi-dimensional planning (especially in spatio-temporal reasoning and budget compliance), and agentic reasoning strategies offer no consistent improvement. Concisely, TravelEval facilitates travel plan evaluation via grounded spatio-temporal emulation and comprehensive metrics, providing a robust foundation for advancing LLM-powered travel planning research and applications.

Problem

• travel planning
• Large Language Models
• benchmarking
• evaluation framework
• spatio-temporal reasoning

Innovation

• TravelEval
• LLM-powered travel planning
• six-dimensional evaluation
• realistic data sandbox
• simulation-based global evaluation