UniData: Universal Multimodal Instruction Generation Pipeline

2026-10-08

AI Summary

This study addresses the high construction costs, limited modality coverage, and difficulties in multi-turn generation associated with multimodal instruction data. To overcome these challenges, we propose a universal data generation pipeline that leverages any-to-any large models alongside event decomposition techniques to transform simple requirements into multi-turn multimodal instructions. Furthermore, a turn-correlation-based quality enhancement algorithm is introduced to rectify redundant reasoning. Using this framework, we construct UniDataset, which encompasses nine distinct modalities and achieves state-of-the-art data quality. Extensive evaluations demonstrate that UniDataset significantly enhances both the comprehension and generation capabilities of downstream multimodal models, establishing it as a robust resource for advancing multimodal instruction tuning.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs remains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round instructions. To address these problems, we introduce UniData, a universal instruction generation pipeline, to transform simple user requirements into multi-round, multimodal instructions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruction generation. Finally, UniData enhances data quality by correcting irrelevant and redundant inference flow, leveraging correlations between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising 20,000 entries across nine modalities for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation capabilities of other multimodal models.

Problem

• Multimodal Large Language Models
• Instruction Dataset
• Data Generation
• Multi-round Instructions
• Modality Support

Innovation

• Multimodal Instruction Generation
• Any-to-Any Model
• Multi-round Dialogue
• Data Quality Enhancement
• Universal Pipeline