How Often Should a Recommender Call an LLM? Value-Weighted Routing, Monitoring, and Seasonal Robustness

2026-07-27

AI Summary

This work addresses a critical limitation in traditional recommendation systems that route queries between lightweight heuristics and large language models solely based on task difficulty, ignoring disparities in error cost and business value. To overcome this, the authors propose a value-weighted routing mechanism that makes unsupervised routing decisions by jointly estimating task difficulty and item-level business value within a fully synthetic retail assortment simulation environment. The framework incorporates decision logging and monitoring modules to uncover category-level biases obscured by aggregate metrics. Through slow-path budget control and seasonal parameter tuning, the system achieves a 60% recall rate on high-value items while improving overall accuracy from 94.3% to 98.3%, demonstrating enhanced robustness under simulated Black Friday traffic surges.

Abstract

Routing decisions between a cheap heuristic and an expensive large language model (LLM) are typically framed as a difficulty problem: send the hard cases to the expensive path. We argue this framing is incomplete because difficulty and business value are distinct axes - a difficult cheap item and a difficult costly item do not have the same cost of error. We present Value Router, a fully synthetic simulation of a retail merchandising pipeline that routes items using only estimated difficulty and estimated value, never ground truth. The study has three stages. First, a value-weighted threshold router is compared with a difficulty-only and a random baseline on a synthetic catalog with an inverse correlation between category volume and value. Value-weighting matches the difficulty-only baseline's recall of true high-value items (60%) while achieving substantially higher precision (98.3% vs. 94.3%). Second, a decision logger and monitor expose a failure mode hidden by aggregate metrics showing that the aggregate result is driven almost entirely by between-category differences rather than per-item discrimination. Third, a simulated Black Friday demand surge (2.5 volume with a shift toward higher-value categories) compares a static router, a seasonally tuned router, and two slow-path budget policies. All results are from a controlled synthetic simulation with experimenter-defined ground truth and illustrate design principles for cost-aware routing systems rather than validated real-world claims.

Problem

• recommender systems
• large language models
• cost-aware routing
• value-weighted decision
• LLM routing

Innovation

• value-weighted routing
• LLM routing
• synthetic simulation
• cost-aware recommendation
• seasonal robustness