Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

2026-08-06

AI Summary

This work addresses the limitations of existing evaluations for financial large language models, which predominantly rely on static question-answering or final profit-and-loss outcomes while neglecting personalized investment reasoning. To overcome this, the authors introduce InvestLogicBench, a novel benchmark constructed from over 200,000 decision trajectories of 151 real investors, establishing a structured evaluation paradigm—Profile → Event → Reasoning → Decision → Outcome (P→E→R→D→O)—that enables traceable assessment of an agent’s logical consistency, event grounding, and reasoning quality. Experiments reveal that while mainstream large models achieve relatively high scores in logical coherence (≈4/5), they exhibit markedly poor event grounding (0.8–2.8/5), and their profitability shows little correlation with reasoning quality, thereby exposing the inadequacy of purely outcome-based evaluation frameworks.

Abstract

Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries. Yet financial LLMs are evaluated either by static question answering or by terminal profit and loss. The former omits agency; the latter cannot reveal whether a profitable action was grounded, profile-consistent, or merely lucky. We ask whether the community is using the wrong ruler for consequential agents. We introduce \textsc{InvestLogicBench}, a process-native benchmark containing 201,247 documented decisions from 151 real-world investors. Each episode instantiates a \textbf{P$\rightarrow$E$\rightarrow$R$\rightarrow$D$\rightarrow$O} trace: investor \textit{Profile}, observable market \textit{Events}, investment \textit{Reasoning}, executable \textit{Decision}, and delayed \textit{Outcome}. The release includes profile construction, point-in-time event binding, structured logic, horizons, outcomes, and post-mortems, and supports comprehension, profile-conditioned generation, and end-to-end replay. Across four leading LLMs, logical plausibility remains near 4/5 while event grounding is only 0.8--2.8/5; return and process quality also disagree. These results expose polished but weakly grounded reasoning that outcome-only evaluation hides. We further argue that P$\rightarrow$E$\rightarrow$R$\rightarrow$D$\rightarrow$O should be a data-system interface, requiring versioned profiles, temporal provenance, inspectable retrieval, decision ledgers, and replayable outcomes. Finance is our stress test for a broader class of personalized, consequential agents.

Problem

• investment competence
• personalized financial agents
• evaluation benchmark
• reasoning grounding
• consequential agents

Innovation

• InvestLogicBench
• personalized financial agents
• process-native benchmark
• grounded reasoning
• P→E→R→D→O trace