Selected work

Representative Papers

Improved Evidence Extraction for Document Inconsistency Detection with LLMs

Jan 06, 2026arXiv.org

This work addresses the limited accuracy of evidence extraction in large language models for document inconsistency detection. To overcome the shortcomings of conventional direct prompting, the authors propose the “Red-Delete-Retry” framework coupled with a constraint-based filtering mechanism. A comprehensive evaluation metric is introduced to systematically assess the completeness and reliability of extracted evidence. Experimental results demonstrate that the proposed approach significantly enhances evidence extraction performance, consistently outperforming existing baselines across multiple benchmarks. The method thus provides more robust support for inconsistency detection tasks by improving both the precision and trustworthiness of the retrieved evidence.

1 citationsRead paper

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

Sep 30, 2026

This study addresses the reliance on costly expert annotations, challenges in long-context processing, and prevalent hallucinations lacking trustworthy benchmarks when large language models (LLMs) analyze financial reports. We propose an expert-annotation-free numerical evidence evaluation method. By constructing an automated data pipeline to generate an S&P 500 benchmark dataset, we introduce the first unsupervised groundedness metric and a "conscious incompetence" detection mechanism to identify scenarios with insufficient evidence. Experiments reveal that while LLMs exhibit strong grounding capabilities, their factual correctness remains limited, rendering them prone to hallucination when critical information is missing. This work establishes a novel paradigm for quantifying and mitigating LLM hallucinations in the financial domain.

0 citationsRead paper

Agentic Method for Deterministic Validation of Legacy Code Migration

Jul 30, 2026

Migrating COBOL systems to Java often suffers from inadequate testing methodologies, leading to challenges in ensuring functional equivalence and comprehensive boundary coverage. This work proposes the “Locksmith Loop,” a novel approach that introduces, for the first time, an agent-based test synthesis mechanism grounded in deterministic oracles. By establishing a co-executable environment between the original COBOL and the target Java code, the method enables deep, automated, and high-fidelity validation through runtime instrumentation, branch-guided witness search, parity-preserving mutation, and locked paragraph analysis. Evaluated on three real-world cases—including two open-source projects and an internal production system—the approach achieves near-complete branch coverage in the former and 91.90% in the latter, with all passing test cases rigorously satisfying deterministic functional equivalence.

0 citationsRead paper

Empowering Small Language Models with Factual Hallucination-Aware Reasoning for Financial Classification

Jan 04, 2026arXiv.org

This study addresses the performance degradation of small language models in financial text classification due to factual hallucinations. It is the first to establish a positive correlation between factual hallucinations and misclassification in financial contexts. To mitigate this issue, the authors propose AAAI—a three-stage framework comprising Association Identification, Automated Detection, and Adaptive Inference. The framework first identifies potential hallucinations through association analysis, then employs an encoder-based factual verifier to automatically detect erroneous claims, and finally leverages a feedback-driven adaptive inference mechanism to guide the model toward corrected predictions. Experiments on three representative small language models demonstrate that the proposed approach effectively detects and alleviates hallucinations, leading to significant improvements in financial classification accuracy.

0 citationsRead paper

AutoMetrics: Approximate Human Judgements with Automatically Generated Evaluators

Dec 19, 2025

Human evaluation of open-domain AI applications—such as travel planning and clinical note generation—suffers from sparse feedback, high latency, and prohibitive costs. To address this, we propose MetricBank, a low-data-dependency automatic evaluation framework that pioneers a joint “retrieval-augmented + LLM-as-a-judge” modeling paradigm: it retrieves highly relevant candidate metrics from the MetricBank metric repository and fine-tunes a lightweight multi-metric regression model using minimal human feedback (<100 annotations), yielding an interpretable, human-aligned surrogate reward. Evaluated across five diverse open-domain tasks, MetricBank achieves up to 33.4% higher Kendall correlation with human judgments than standalone LLM-based evaluators, significantly outperforming existing automated evaluation approaches. We publicly release the full toolkit alongside MetricBank—a curated repository of 48 high-quality, domain-agnostic evaluation metrics.

0 citationsRead paper

Recent publications

Latest Papers

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

Sep 30, 2026

This study addresses the reliance on costly expert annotations, challenges in long-context processing, and prevalent hallucinations lacking trustworthy benchmarks when large language models (LLMs) analyze financial reports. We propose an expert-annotation-free numerical evidence evaluation method. By constructing an automated data pipeline to generate an S&P 500 benchmark dataset, we introduce the first unsupervised groundedness metric and a "conscious incompetence" detection mechanism to identify scenarios with insufficient evidence. Experiments reveal that while LLMs exhibit strong grounding capabilities, their factual correctness remains limited, rendering them prone to hallucination when critical information is missing. This work establishes a novel paradigm for quantifying and mitigating LLM hallucinations in the financial domain.

0 citationsRead paper

Agentic Method for Deterministic Validation of Legacy Code Migration

Jul 30, 2026

Migrating COBOL systems to Java often suffers from inadequate testing methodologies, leading to challenges in ensuring functional equivalence and comprehensive boundary coverage. This work proposes the “Locksmith Loop,” a novel approach that introduces, for the first time, an agent-based test synthesis mechanism grounded in deterministic oracles. By establishing a co-executable environment between the original COBOL and the target Java code, the method enables deep, automated, and high-fidelity validation through runtime instrumentation, branch-guided witness search, parity-preserving mutation, and locked paragraph analysis. Evaluated on three real-world cases—including two open-source projects and an internal production system—the approach achieves near-complete branch coverage in the former and 91.90% in the latter, with all passing test cases rigorously satisfying deterministic functional equivalence.

0 citationsRead paper

Improved Evidence Extraction for Document Inconsistency Detection with LLMs

Jan 06, 2026arXiv.org

This work addresses the limited accuracy of evidence extraction in large language models for document inconsistency detection. To overcome the shortcomings of conventional direct prompting, the authors propose the “Red-Delete-Retry” framework coupled with a constraint-based filtering mechanism. A comprehensive evaluation metric is introduced to systematically assess the completeness and reliability of extracted evidence. Experimental results demonstrate that the proposed approach significantly enhances evidence extraction performance, consistently outperforming existing baselines across multiple benchmarks. The method thus provides more robust support for inconsistency detection tasks by improving both the precision and trustworthiness of the retrieved evidence.

1 citationsRead paper

Empowering Small Language Models with Factual Hallucination-Aware Reasoning for Financial Classification

Jan 04, 2026arXiv.org

This study addresses the performance degradation of small language models in financial text classification due to factual hallucinations. It is the first to establish a positive correlation between factual hallucinations and misclassification in financial contexts. To mitigate this issue, the authors propose AAAI—a three-stage framework comprising Association Identification, Automated Detection, and Adaptive Inference. The framework first identifies potential hallucinations through association analysis, then employs an encoder-based factual verifier to automatically detect erroneous claims, and finally leverages a feedback-driven adaptive inference mechanism to guide the model toward corrected predictions. Experiments on three representative small language models demonstrate that the proposed approach effectively detects and alleviates hallucinations, leading to significant improvements in financial classification accuracy.

0 citationsRead paper

AutoMetrics: Approximate Human Judgements with Automatically Generated Evaluators

Dec 19, 2025

Human evaluation of open-domain AI applications—such as travel planning and clinical note generation—suffers from sparse feedback, high latency, and prohibitive costs. To address this, we propose MetricBank, a low-data-dependency automatic evaluation framework that pioneers a joint “retrieval-augmented + LLM-as-a-judge” modeling paradigm: it retrieves highly relevant candidate metrics from the MetricBank metric repository and fine-tunes a lightweight multi-metric regression model using minimal human feedback (<100 annotations), yielding an interpretable, human-aligned surrogate reward. Evaluated across five diverse open-domain tasks, MetricBank achieves up to 33.4% higher Kendall correlation with human judgments than standalone LLM-based evaluators, significantly outperforming existing automated evaluation approaches. We publicly release the full toolkit alongside MetricBank—a curated repository of 48 high-quality, domain-agnostic evaluation metrics.

0 citationsRead paper