CASPER-Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based systems

2026-07-31

AI Summary

This work addresses the challenges in regression testing of large language model (LLM) systems, where individual test instances often lack sufficient sensitivity while aggregate metrics are too coarse-grained. To overcome this, the paper introduces CASPER, a novel framework that uniquely integrates slice-level analysis with change-aware mechanisms. CASPER constructs semantically coherent and behaviorally consistent test slices through evolutionary slicing and prioritizes them based on execution logs according to their likelihood of revealing regressions. This approach significantly enhances the granularity and efficiency of regression detection under both prompt-level and model-level changes. Experimental results demonstrate that CASPER-generated slices maintain strong semantic coherence while achieving higher behavioral consistency, and its regression-aware slice prioritization consistently outperforms baseline methods across diverse change types and testing budgets.

Abstract

Regression testing for LLM-based systems poses unique challenges because individual regression instances provide limited information about system-level regressions. A failure in a single instance does not necessarily indicate a meaningful regression or provide sufficient information to diagnose affected behaviors. Conversely, detecting regressions based only on overall system performance changes is too coarse-grained, as it does not identify which behaviors are affected. This motivates analyzing regression instances at an intermediate level through test suite slices. To address this challenge, we propose CASPER, a change-aware slice prioritization framework for efficient regression testing of prompt-level and model-level changes in LLM-based systems. CASPER first identifies regression slices containing semantically related instances with consistent performance characteristics using an evolutionary slice identification approach. Given a change to an LLM-based application, CASPER prioritizes slices according to their likelihood of regression using behavioral information extracted from execution logs. We instantiate CASPER in the software issue resolution domain and evaluate slice identification against clustering-based baselines and regressed slice prioritization against a random ranking baseline. Results show that CASPER generates more consistent slices while maintaining comparable or better semantic coherence and improves regressed slice prioritization across different LLM-based system changes and testing budgets.

Problem

• regression testing
• LLM-based systems
• test suite slices
• behavioral regression
• change-aware prioritization

Innovation

• change-aware
• slice prioritization
• regression testing
• LLM-based systems
• evolutionary slice identification