FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering

2026-09-20

AI Summary

本文针对金融问题中的歧义解决，提出FinInteract基准，通过提供默认和预期解释来评估模型是否能正确澄清并整合意图。

Abstract

Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specified: Meta Platforms' "operating income" is $46.75B consolidated but $62.87B for the Family of Apps segment, and each reading is exactly verifiable against the filing. A capable agent should recognize the ambiguity and ask, rather than commit to a plausible but unintended reading. Existing financial benchmarks cannot measure this, because one gold answer per question cannot separate agents that resolve the ambiguity from those that guess the common reading, a blind spot we call the single-gold illusion. We release FinInteract, a bilingual (English/Chinese) benchmark of 173 instances that pairs each question with a default and an intended interpretation across a five-category ambiguity taxonomy, and grades whether an agent elicits the right clarification and then integrates it. Re-grading identical outputs against the default rather than the intended reading inflates GPT-4o's accuracy by 3.1 times, confirming the illusion. Beyond it, we find that models answer above 90% once the interpretation is supplied but at most 28.9% when they must elicit it themselves, that targeting is uneven across a taxonomy well powered for entity scope and metric definition and exploratory elsewhere, and that conditioning on the ambiguity category improves resolution at both inference and training time.

Problem

• Ambiguity
• Financial Question Answering
• Clarification
• Large Language Models
• Benchmark

Innovation

• Clarification Request
• Intent Integration
• Ambiguity Resolution
• Single-Gold Illusion
• Bilingual Benchmark