Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?

2026-07-29

AI Summary

This study evaluates the financial numerical reasoning capabilities of large language models in real-world credit card agreement scenarios, with a focus on their limitations in understanding consumer-critical aspects such as fees, interest rates, and repayment terms. We introduce CreditCardQA, the first structured question-answering dataset derived from actual credit card contracts, comprising 1,800 questions, and establish the first benchmark by translating real financial clauses into machine-reasonable formats. Leveraging Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting, augmented with error analysis and difficulty stratification, we find that model errors predominantly stem from misapplication of rules and misinterpretation of clauses rather than arithmetic mistakes. PoT substantially enhances weaker models, narrowing the performance gap between open- and closed-source systems, particularly excelling in conditional logic, monetary constraints, and comparative queries. The work also highlights systematic failures in edge cases like late-payment penalties, which may disproportionately affect low-income consumers.

Abstract

We introduce CreditCardQA, the first financial literacy benchmark for numerical reasoning derived from real credit card agreements. The dataset contains 1,800 questions, including first-person variants that reflect how consumers naturally ask about fees, interest, and payments. We evaluate a range of large language and reasoning models under Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting. Overall, PoT yields consistent performance gains, particularly for models with weaker baseline reasoning, and narrows gaps between open- and closed-source systems. Through error analysis, we show that failures arise less from arithmetic and more from misapplied financial rules, missed conditions, and misunderstandings of contractual terms. We further analyze question difficulty and find that comparisons, conditional logic, and monetary constraints are especially challenging. We also find that errors often arise in edge cases such as late-payment penalties or small-balance scenarios that are more likely to affect lower-income or financially vulnerable individuals.

Problem

• financial literacy
• numerical reasoning
• language model reasoning
• credit card agreements
• conditional logic

Innovation

• CreditCardQA
• numerical reasoning
• Program-of-Thought
• financial literacy
• error analysis