LCTG Bench: LLM Controlled Text Generation Benchmark

2025-01-27

AI Summary

Existing evaluation frameworks lack Japanese-specific benchmarks for assessing controllability of large language models (LLMs), hindering systematic comparison of instruction-following capabilities between multilingual and Japanese-specialized models. Method: We introduce JControlBench—the first Japanese-controllable text generation benchmark—featuring a linguistically adapted instruction control task suite and a unified, cross-task controllability evaluation framework. It covers nine state-of-the-art Japanese and multilingual LLMs, including GPT-4, and employs standardized, reproducible quantitative metrics. Contribution/Results: Our experiments provide the first systematic evidence that Japanese-specialized models significantly outperform multilingual counterparts in fine-grained instruction execution and cultural-context adaptation. JControlBench fills a critical gap in controllability evaluation for low-resource languages and establishes a rigorous, reliable foundation for Japanese LLM development, selection, and deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Intelligent Robots: Behavior Learning & Control

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructions. In response, benchmarks focusing on the controllability of LLMs have been developed, but several issues remain: (1) They primarily cover major languages like English and Chinese, neglecting low-resource languages like Japanese; (2) Current benchmarks employ task-specific evaluation metrics, lacking a unified framework for selecting models based on controllability across different use cases. To address these challenges, this research introduces LCTG Bench, the first Japanese benchmark for evaluating the controllability of LLMs. LCTG Bench provides a unified framework for assessing control performance, enabling users to select the most suitable model for their use cases based on controllability. By evaluating nine diverse Japanese-specific and multilingual LLMs like GPT-4, we highlight the current state and challenges of controllability in Japanese LLMs and reveal the significant gap between multilingual models and Japanese-specific models.

Problem

• Controllability
• Japanese Language
• Large Language Models

Innovation

• LCTG Bench
• Multilingual Model Evaluation
• Japanese Language Modeling