Improving Cross-Format Robustness in Language Models with Multi-Format Training

2026-06-10

AI Summary

This work addresses the sensitivity of large language models to answer formatting, where semantically equivalent questions yield performance fluctuations due to superficial format differences. To enhance cross-format robustness, the authors propose FormatMix, a lightweight training strategy that augments approximately 30% of the training data with multiple answer formats through a combination of random and target-oriented augmentation. Experimental results demonstrate that FormatMix consistently improves model performance across both GLM-4 and Llama-3.1, achieving near-optimal results comparable to full-format supervision with only limited multi-format labeled data. Moreover, it significantly outperforms training solely on multiple-choice formats, underscoring the critical role of format diversity in boosting task performance and output consistency.

Abstract

Large language models often remain sensitive to answer format: a question solved correctly in one form may fail in another semantically equivalent form. To study this gap, we define cross-format robustness as the extent to which a model answers the same underlying question consistently across formats. We then compare full-format training with FormatMix, which expands only a subset of training items into multiple equivalent formats using either random or targeted selection. Across GLM4 and Llama-3.1, multi-format supervision consistently improves both task performance and cross-format robustness, whereas Multiple-choice question (MCQ)-only supervision alone brings little benefit and can even reduce robustness. We further find that expanding only about 30% of the training set into multiple formats often recovers most of the gain from full-format training, and this effect appears across the model families and sizes we study. These results suggest that format diversity, rather than additional supervision alone, is the key driver of robustness. That lightweight multi-format augmentation is a practical way to make LLMs less sensitive to answer format without changing the base model.

Problem

• cross-format robustness
• language models
• answer format
• format sensitivity
• semantic equivalence

Innovation

• cross-format robustness
• multi-format training
• FormatMix
• format diversity
• language model robustness