SemSketches-2021: experimenting with the machine processing of the pilot semantic sketches corpus

2025-05-23

AI Summary

This study addresses the machine understanding of semantic sketches—i.e., modeling and matching anonymized, predicate-driven semantic structures. To this end, we first propose a formal representational framework for semantic sketches, construct and publicly release the first open-annotated corpus, and design a supervised evaluation task based on contextual matching. Our methodology integrates semantic role labeling, dependency parsing, and cross-sentence alignment to support standardized, multi-model evaluation. The work advances predicate-centric semantic structure modeling: it empirically validates the efficacy of semantic sketches in predicate disambiguation and semantic role recovery; top-performing systems achieve matching accuracies of 68–82%; and it establishes a reproducible, extensible benchmark for corpus construction and evaluation. This foundation facilitates future research in semantic computation, offering both theoretical insight and practical infrastructure for modeling predicate-level meaning.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Data Mining & Knowledge Management: Semantic Web
• Machine Learning: Learning Preferences or Rankings

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The paper deals with elaborating different approaches to the machine processing of semantic sketches. It presents the pilot open corpus of semantic sketches. Different aspects of creating the sketches are discussed, as well as the tasks that the sketches can help to solve. Special attention is paid to the creation of the machine processing tools for the corpus. For this purpose, the SemSketches-2021 Shared Task was organized. The participants were given the anonymous sketches and a set of contexts containing the necessary predicates. During the Task, one had to assign the proper contexts to the corresponding sketches.

Problem

• Developing machine processing methods for semantic sketches
• Creating an open corpus of pilot semantic sketches
• Designing tools to match contexts with semantic sketches

Innovation

• Machine processing of semantic sketches corpus
• Pilot open corpus for semantic sketches
• Shared Task for context-sketch matching