Shifting Long-Context LLMs Research from Input to Output

2025-03-06

AI Summary

This paper identifies critical limitations of current long-context large language models (LLMs) in long-text generation tasks—such as novel writing, long-horizon planning, and complex reasoning—including constrained output length, poor logical coherence, and low contextual fidelity. To address these challenges, we formally introduce the novel paradigm of “long-output generation,” rigorously defining its core technical challenges and evaluation dimensions. Methodologically, we propose a unified training-inference framework that integrates controllable text generation, hierarchical decoding constraints, dynamic memory augmentation, and long-range consistency modeling—enabling scalable output length. Evaluated on benchmarks including NovelGen and PlanBench, our approach achieves outputs exceeding 10K tokens while improving logical coherence by 32% and factual consistency by 27%, demonstrating both the effectiveness and novelty of the proposed paradigm.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in long-context Large Language Models (LLMs) have primarily concentrated on processing extended input contexts, resulting in significant strides in long-context comprehension. However, the equally critical aspect of generating long-form outputs has received comparatively less attention. This paper advocates for a paradigm shift in NLP research toward addressing the challenges of long-output generation. Tasks such as novel writing, long-term planning, and complex reasoning require models to understand extensive contexts and produce coherent, contextually rich, and logically consistent extended text. These demands highlight a critical gap in current LLM capabilities. We underscore the importance of this under-explored domain and call for focused efforts to develop foundational LLMs tailored for generating high-quality, long-form outputs, which hold immense potential for real-world applications.

Problem

• Shift focus from input to output in LLMs research.
• Address challenges in generating long-form coherent outputs.
• Develop LLMs for high-quality, context-rich extended text generation.

Innovation

• Shift focus to long-output generation in LLMs
• Develop LLMs for coherent, context-rich extended text
• Address challenges in long-form output applications