Evaluating and Pricing Advertisements in AI-Generated Responses

2026-07-30

AI Summary

This work addresses the challenges of evaluating click-through intent and establishing fair pricing for advertisements embedded in large language model (LLM)-generated content, which stem from scarce behavioral logs, unreliable human annotations, and the conflation of intent with textual fluency in existing models. To overcome these limitations, the authors propose a psychology-inspired agent simulation framework that generates synthetic supervision signals and distills them into a parameter-efficient, differentiable evaluator capable of continuous click-intent estimation without real user interaction data. Integrating mechanism design theory, they further derive a unique payment rule that guarantees truthful bidding as the optimal strategy and supports non-monotonic allocation. Experiments demonstrate that the proposed evaluator significantly outperforms state-of-the-art zero-shot baselines in relevance sensitivity (79%), cross-product generalization (zero error across 103 fictional products), and alignment with human preferences (86% pairwise agreement).

Abstract

As search increasingly shifts toward LLM-driven answer engines, advertising is becoming embedded within the generated response itself and should therefore be evaluated for both user utility and commercial value. The key challenge is click-through intent: behavioural logs are unavailable, human annotation resists calibration, and frontier LLM judges conflate intent with linguistic fluency. These gaps compound, as principled pricing presupposes a continuous intent signal, while generating such a signal presupposes supervision that is currently unavailable. We construct the missing supervision through a psychologically grounded agent simulation framework, and distil it into a parameter-efficient evaluator that predicts click-through intent, together with the three companion dimensions of ad quality, as smooth, differentiable estimates. Validated through sign-certain behavioural perturbations, the evaluator surpasses frontier zero-shot judges on relevance sensitivity (79% versus 60-67%), tracks graded content degradation, generalises without error to 103 fictional products, and agrees with human preference in 86% of pairwise judgements across five annotators, with agreement rising in the evaluator's confidence. Upon its estimates we build the pricing layer directly, deriving the unique payment rule under which truthful bidding is optimal, demonstrating it on a best-of-k allocation, and extending the mechanism to non-monotone allocations. The same differentiable signal stands ready as a training objective for ad generation.

Problem

• click-through intent
• advertising evaluation
• LLM-generated responses
• ad pricing
• supervision gap

Innovation

• click-through intent
• agent simulation
• differentiable evaluator
• ad pricing
• LLM-based advertising