UltraIF: Advancing Instruction Following from the Wild

2025-02-06

AI Summary

Open-source large language models (LLMs) often exhibit weaker instruction-following capabilities compared to state-of-the-art closed-source models. Method: This paper proposes UltraIF, a scalable framework that first decomposes real-world user instructions into query–constraint–evaluation triples; it then trains UltraComposer to generate constraint-augmented prompts and jointly employs dual-role (generation + evaluation) models for response filtering and alignment. Contribution/Results: Crucially, UltraIF achieves zero-shot self-alignment to the performance level of an instruction-tuned model—using only the 8B base model LLaMA-3.1-8B-Base, without instruction fine-tuning or reliance on any benchmark data. Experiments demonstrate that UltraIF matches LLaMA-3.1-8B-Instruct across five major instruction-following benchmarks in zero-shot settings; it further attains competitive zero-shot transfer performance on unseen benchmarks, validating its generalizability and practical utility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Instruction-following made modern large language models (LLMs) helpful assistants. However, the key to taming LLMs on complex instructions remains mysterious, for that there are huge gaps between models trained by open-source community and those trained by leading companies. To bridge the gap, we propose a simple and scalable approach UltraIF for building LLMs that can follow complex instructions with open-source data. UltraIF first decomposes real-world user prompts into simpler queries, constraints, and corresponding evaluation questions for the constraints. Then, we train an UltraComposer to compose constraint-associated prompts with evaluation questions. This prompt composer allows us to synthesize complicated instructions as well as filter responses with evaluation questions. In our experiment, for the first time, we successfully align LLaMA-3.1-8B-Base to catch up with its instruct version on 5 instruction-following benchmarks without any benchmark information, using only 8B model as response generator and evaluator. The aligned model also achieved competitive scores on other benchmarks. Moreover, we also show that UltraIF could further improve LLaMA-3.1-8B-Instruct through self-alignment, motivating broader use cases for the method. Our code will be available at https://github.com/kkk-an/UltraIF.

Problem

• Bridging gap between open-source and company-trained LLMs
• Enhancing complex instruction-following in LLMs
• Improving LLaMA-3.1-8B-Instruct through self-alignment

Innovation

• Decomposes user prompts
• Trains UltraComposer for prompts
• Aligns LLaMA-3.1-8B-Base