Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models

2026-03-10

AI Summary

This study addresses the lack of systematic evaluation of large language models’ (LLMs) reasoning capabilities in quantitative investment decision-making, as existing financial benchmarks primarily focus on language understanding. The authors propose the first scalable multiple-choice benchmark derived from portfolio optimization problems with analytical solutions, grounded in modern portfolio theory. By parameterizing objective functions and constraints, they generate a diverse test set that directly assesses LLMs’ ability to reason about financial optimization tasks. Experimental results reveal distinct performance patterns: GPT-4 demonstrates superior and robust performance under risk-oriented objectives, Gemini 1.5 Pro excels in return-focused scenarios but is sensitive to constraint variations, and Llama 3.1-70B exhibits comparatively weaker overall performance. These findings highlight both the disparities and limitations of current LLMs in quantitative financial reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This study introduces a benchmark framework for evaluating the financial decision-making capabilities of large language models (LLMs) through portfolio optimization problems with mathematically explicit solutions. Unlike existing financial benchmarks that emphasize language-processing tasks, the proposed framework directly tests optimization-based reasoning in investment contexts. A large set of multiple-choice questions is generated by varying objectives, candidate assets, and investment constraints, with each problem designed to include a unique correct solution and systematically constructed alternatives. Experimental results comparing GPT-4, Gemini 1.5 Pro, and Llama 3.1-70B reveal distinct performance patterns: GPT achieves the highest accuracy in risk-based objectives and remains stable under constraints, Gemini performs well in return-based tasks but struggles under other conditions, and Llama records the lowest overall performance. These findings highlight both the potential and current limitations of LLMs in applying quantitative reasoning to finance, while providing a scalable foundation for developing LLM-based services in portfolio management.

Problem

• portfolio optimization
• large language models
• financial decision-making
• benchmark framework
• quantitative reasoning

Innovation

• portfolio optimization
• large language models
• financial benchmarking
• quantitative reasoning
• constrained optimization