Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

2026-03-16

AI Summary

This work addresses the lack of effective evaluation methods for assessing large language models’ understanding of cross-sentential systematic linguistic phenomena, such as verb alternations. To this end, the authors construct the first multilingual verb alternation dataset covering English, German, Italian, and Hebrew. Leveraging the Blackbird Language Matrix (BLM) framework, they design controlled linguistic puzzles using three syntactically and semantically structured templates of increasing complexity. The dataset integrates both synthetic and naturally occurring examples through linguistically informed data augmentation. This resource not only fills a critical gap in evaluating cross-sentential systematic linguistic knowledge but also provides baseline results across the four languages, demonstrating its diagnostic value and effectiveness in probing the linguistic capabilities of large language models.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Large language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems. The BLM task -- an RPM/ARC-like task devised specifically for language -- is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.

Problem

• verb alternations
• paradigm-based datasets
• cross-sentence patterns
• large language models
• linguistic evaluation

Innovation

• verb alternations
• Blackbird Language Matrices
• paradigm-based datasets
• data augmentation
• cross-lingual probing