Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents

2026-07-16

AI Summary

This work addresses the complexity of payment integration in warehouse-scale software development, which entails product selection, end-to-end workflow implementation, and ensuring state consistency. The study introduces the first structured evaluation benchmark tailored to real-world Alipay payment scenarios, encompassing nine products and eighteen tasks categorized into basic functionality and advanced risk-control scenarios. A multidimensional evaluation framework is developed, integrating static analysis, unit, integration, and end-to-end testing, along with large language model–assisted semantic assessment. Experimental results demonstrate that incorporating payment-specific skills improves the average pass rate of six coding agents by 10.31 percentage points, achieving performance ranging from 68.58% to 91.37%, thereby validating the benchmark’s effectiveness and practicality in fine-grained diagnosis of agent capabilities in payment integration.

Abstract

Payment integration is a demanding repository-level software task: agents must select a suitable product, implement coordinated client-server flows, verify payment outcomes, and preserve consistency between transaction and business states. We introduce Alipay-PIBench, a benchmark for evaluating coding agents on realistic Alipay payment integration. It contains nine product-specific projects and 18 task instances, each organized into Basic functional-completion and Advanced risk-aware hardening scenarios. Scenario-specific rubrics support deterministic static, unit, integration, and end-to-end checks, supplemented by LLM-assisted assessment for semantic requirements. We evaluate six coding-agent models and report rubric pass rate (RPR). Under the with-skill condition, mean RPR ranges from 68.58% to 91.37%. Access to the alipay-payment-integration skill improves mean RPR by 10.31 percentage points on average relative to the without-skill condition, with gains varying across models, products, and scenarios. Method-level results distinguish source-level completion, executable payment behavior, and payment-domain requirements. Alipay-PIBench provides a controlled setting for diagnosing model capability and evaluating structured guidance in payment integration.

Problem

• payment integration
• coding agents
• software benchmark
• Alipay
• transaction consistency

Innovation

• payment integration
• coding agents
• benchmark
• risk-aware hardening
• rubric-based evaluation