Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models

2025-01-09

AI Summary

Large language models (LLMs) struggle to reliably adhere to *soft constraints*—semantically rich, human-intended requirements that lack automated verifiability—hindering their trustworthy deployment in real-world applications. To address this, we propose the first systematic framework for soft constraint adherence: (1) a constraint-aware curriculum learning paradigm that incrementally increases semantic complexity during training; (2) a fully automated, annotation-free pipeline for generating high-quality soft constraint data; and (3) a constraint-driven prompt optimization and evaluation protocol. Our approach achieves significant improvements in constraint adherence across multiple soft-constraint benchmarks. Ablation studies confirm the critical roles of both the curriculum strategy and data quality. All generated data, implementation code, and evaluation protocols are publicly released to foster reproducibility and further research.

Technology Category

Constraint Satisfaction and Optimization: Constraint Learning and AcquisitionMachine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Safety and Robustness

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, soft constraints are semantically related and difficult to verify through automated methods. These constraints remain a significant challenge for LLMs. To enhance the ability of LLMs to follow soft constraints, we initially design a pipeline to obtain high-quality outputs automatically. Additionally, to fully utilize the acquired data, we introduce a training paradigm based on curriculum learning. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements. The datasets and code are publicly available at https://github.com/Rainier-rq/FollowSoftConstraints.

Problem

• Large Language Models
• Soft Constraints
• Compliance

Innovation

• Automatic Pipeline
• Curriculum Learning
• Soft Rules Compliance