Diversity-Guided Search-Based Testing of Large Language Model Applications

2026-09-19

AI Summary

为解决大型语言模型应用中的不准确、虚构或有害响应问题，本文提出一种基于搜索的测试框架，该框架以失败多样性为优化目标，并通过存档生成测试和重新填充操作来维持探索。

Abstract

Large Language Model (LLM)-based applications are increasingly deployed across domains including customer service, education, and mobility. These systems are prone to inaccurate, fictitious, or harmful responses, and their vast, high-dimensional input space makes systematic testing particularly challenging. In this paper, we present a search-based testing framework for LLM-based applications that incorporates failure diversity as an explicit optimization objective. Building on a discretization along stylistic, content-related, and perturbation dimensions, our framework maintains an archive of generated tests and rewards distance from that archive, while a repopulation operator periodically replaces non-failing tests in the population to sustain exploration. The repopulation operator is parameterized by its sampling strategy: replacement candidates are drawn either uniformly or by greedy distance maximization, two settings we compare empirically. We evaluate both against several baselines across three case studies---LLM safety, in-car navigation, and in-vehicle function control---covering five systems under test, eight LLMs, and 18 distinct test configurations, with over one million executed tests. Our results show that all guided variants detect substantially more failures than random and combinatorial search in nearly all configurations. Among them, diversified search detects fewer failures, but covers a broader range of failure types in all case studies, with greedy repopulation offering the best tradeoff.

Problem

• Large Language Model
• systematic testing
• failure diversity
• input space
• inaccurate responses

Innovation

• failure diversity
• search-based testing
• repopulation operator
• large language model applications