🤖 AI Summary
This work addresses the critical yet underexplored challenge of aligning prompt templates with the pretraining objectives of language models for knowledge generation tasks, where performance heavily depends on such alignment yet lacks effective guidance for template selection. To bridge this gap, the paper introduces the Maskedness Index (MI)—a novel metric that quantifies the degree of alignment between a given task and the model’s pretraining objective. Built upon the DepthRank algorithm, MI evaluates template suitability by measuring the discrepancy in knowledge relational scores between masked and prefix-style prompting formats. The proposed approach offers both theoretical grounding and a practical tool for prompt engineering, demonstrating on the ATOMIC2020 benchmark a significant positive correlation between MI and downstream generation performance, particularly yielding substantial gains in low-resource knowledge extraction scenarios.
📝 Abstract
Large-scale pretrained language models such as T5 and BERT have demonstrated strong capabilities for generating structured knowledge. However, their performance depends on how closely the prompting strategy matches the objectives used during pretraining. We introduce the Maskability Index (MI), a quantitative metric that estimates whether a knowledge relation is better suited to masked-style prompting or prefix-style prompting in few-shot generation. MI is computed from differences in DepthRank scores between masked and unmasked templates, providing a principled measure of objective-template alignment. We evaluate MI on a diverse set of relations from the ATOMIC2020 knowledge base completion benchmark and show that it is positively correlated with downstream generation performance. These results indicate that MI can help select appropriate prompting templates and adaptation strategies for extracting relational knowledge from pretrained language models, especially in low-resource settings.