Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

2026-07-29

AI Summary

This work addresses the lack of systematic evaluation of behavioral robustness in large language model (LLM) document workflows when confronted with semantically equivalent inputs in varying formats. The authors propose the first format-aware metamorphic testing framework, leveraging three classes of metamorphic relations to conduct a large-scale empirical analysis of end-to-end workflow performance across diverse document formats. They further introduce a lightweight, training-free format adaptation strategy. Across 48,000 experiments, they find that format changes can degrade accuracy by up to 53.63% and induce decision drift in over 41% of instances. Their method recovers up to 44.21% of the lost performance, offering the first systematic evidence of how document formatting impacts LLM reliability and establishing a novel paradigm for testing and mitigating format-induced fragility.

Abstract

LLM-driven software systems are rapidly evolving from plain-text conversations to document-centric end-to-end workflows, where the same semantic content can be delivered in diverse document formats (e.g., CSV) through file upload interfaces. Yet existing testing work focuses on the robustness and reliability of models and systems whose input is a single prompt string, leaving a critical question unanswered: Can these document workflows maintain robust behaviors when the same content arrives in a different document format? To fill the gap, in this paper, we propose a format-aware metamorphic testing framework with three metamorphic relations to comprehensively evaluate the format robustness of end-to-end LLM document workflows. Based on this framework, we conduct a large-scale empirical study spanning four representative LLM workflows, four real-world tasks, and four document formats, comprising a total of 48,000 workflow executions. Our findings reveal that format variation poses a systematic and serious threat. Merely switching formats can cause accuracy to drop by up to 53.63% and trigger decision drifts in over 41% of instances. We further design lightweight mitigation strategies from the users' perspective that recover up to 44.21% of format-induced decision drift without model retraining. Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM software systems, calling for corresponding testing and safeguards in the deployment in real-world high-stakes scenarios.

Problem

• format robustness
• LLM document workflows
• metamorphic testing
• decision drift
• document format

Innovation

• format robustness
• metamorphic testing
• LLM document workflows
• decision drift
• document format