TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning

2026-07-29

AI Summary

This work addresses the limitations of current large language model (LLM) agents in complex itinerary planning, which struggle to simultaneously satisfy multidimensional hard constraints—including budget limits, spatiotemporal feasibility, hallucination-free outputs, and implicit user preferences—and lack a reproducible, auditable evaluation framework. To bridge this gap, we introduce the first end-to-end benchmark for multi-constrained feasible itinerary synthesis, comprising a dataset of structured feasible and infeasible samples, a production-grade RESTful tool sandbox, a rule-based deterministic evaluator, and human-validated gold-standard answers. Evaluating 15 LLM agents across 800 tasks, we find that even the strongest model (GPT-5.6) produces fully feasible itineraries in only 46.2% of solvable tasks, with most models achieving near-zero success rates—highlighting the fulfillment of implicit user needs as the key bottleneck and underscoring the benchmark’s challenge and necessity.

Abstract

Travel planning is a demanding stress test for tool-using LLM agents: a usable itinerary is a single artifact that must be right along many axes at once - every flight, hotel, and attraction must exist and be bookable, the days must be physically traversable, the total must clear a budget, and the plan must serve a traveler whose needs are only partly stated. Existing agent benchmarks reward these properties one at a time and grade the final output with soft or LLM-judged rubrics, which cannot certify that a returned plan is executable and are neither reproducible nor auditable. We introduce TREK (Travel Reasoning and Evaluation Kit), a benchmark for feasible itinerary synthesis: producing a single plan that is jointly constraint-correct, hallucination-free, spatio-temporally executable, budget-valid, and responsive to the traveler's unstated persona needs. TREK comprises 800 multi-constraint tasks - 533 feasible and 267 provably infeasible with typed route/entity/budget causes - over a synthetic, internally consistent knowledge base of 212,530 records across 375 cities and 13 personas, served through a production-style tool sandbox of validated RESTful APIs. Every task is scored by a fully deterministic, rule-based evaluator with no LLM judge and ships a human-verified gold reference that scores a perfect 1.0 under that same evaluator, so the ceiling is demonstrably achievable and every remaining gap is an agent limitation rather than scorer strictness. Evaluating 15 LLM agents across nine constraint dimensions, we find that even the strongest (GPT-5.6) produces a fully-feasible plan on only 46.2% of solvable tasks, with a median of 6.6% and a floor of 0.0%; satisfying travelers' unstated needs emerges as the universal bottleneck, unsolved even at the frontier. We release the dataset, tool sandbox, deterministic evaluator, and agent code as a fully reproducible benchmark.

Problem

• travel planning
• LLM agents
• feasibility evaluation
• itinerary synthesis
• benchmark

Innovation

• feasible itinerary synthesis
• deterministic evaluation
• tool-using LLM agents
• multi-constraint reasoning
• reproducible benchmark