🤖 AI Summary
This work addresses the challenge of selecting effective fine-tuning strategies for encoder-decoder pre-trained language models in generation and question-answering tasks. It proposes the Match Task to Objective (MTO) framework, which establishes the first systematic alignment mechanism between downstream tasks and pre-training objectives. MTO automatically constructs training data and prompt templates that are consistent with the original pre-training objective and extends this alignment to soft prompt tuning, thereby enabling precise task–objective matching. Experimental results demonstrate that MTO achieves over 120% performance improvement under few-shot settings compared to existing methods, significantly outperforms strong baselines in full-data scenarios, and substantially enhances the effectiveness of prompt tuning.
📝 Abstract
Prompt-based learning has emerged as a dominant paradigm in natural language processing. This study explores the impact of diverse pre-training objectives on the performance of encoder-decoder pre-trained language models across generation and question answering tasks, with a focus on commonsense knowledge retrieval and completion. We highlight the benefits of incorporating multiple objectives during both pre-training and fine-tuning stages. We introduce the Match Task to Objective (MTO) framework and methods for determining the appropriate objective for a given task. This framework offers automated methods to prepare task-related data for adaptation through unsupervised training, based on the identified objective. In the fine-tuning stage, we design novel templates that align with the objectives of the pre-training and adaptation stages. When aligned with task requirements, these strategies can achieve a performance gain of over 120\% compared to conventional methods in few-shot settings. They significantly outperform related works in few-shot settings and exceed the baseline even in full-dataset scenarios. Furthermore, we extend this approach to include prompt-tuning methodologies, providing guidance for more effective soft prompt engineering and optimization. Our strategies significantly enhance prompt-tuning performance as well. These insights hold substantial value, precisely guiding the selection and optimization of models customized for specific tasks. Code is available at https://github.com/puraminy/MTO/