MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema

2025-10-18

AI Summary

Existing code datasets are predominantly confined to single-language lexical features or isolated parsers, hindering cross-lingual syntactic reasoning and structural analysis. To address this, we propose a language-agnostic, universal Abstract Syntax Tree (AST) abstraction schema that enables structural alignment and semantic normalization across ten mainstream programming languages. We introduce the first large-scale, high-fidelity multilingual code parsing dataset—comprising over 7 million source files—generated via a unified compilation pipeline and stored in Parquet format, accompanied by reproducibility scripts and interactive visualization tools. The dataset is publicly released on Hugging Face and GitHub. Empirical analysis reveals substantial syntactic structural commonalities across languages, providing foundational support for cross-lingual program understanding, pretraining of code models, and static program analysis.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

We introduce the MultiLang Code Parser Dataset (MLCPD), a large-scale, language-agnostic dataset unifying syntactic and structural representations of code across ten major programming languages. MLCPD contains over seven million parsed source files normalized under our proposed universal Abstract Syntax Tree (AST) schema, enabling consistent cross-language reasoning, structural learning, and multilingual software analysis. Unlike existing corpora that focus purely on token-level code or isolated parsers, MLCPD provides both hierarchical tree representations and rich metadata for every file, ensuring lossless syntactic coverage and structural uniformity. Each entry includes a normalized schema, language-level metadata, and abstracted node semantics stored in Parquet format for scalable retrieval. Empirical analyses reveal strong cross-language structural regularities-demonstrating that syntactic graphs from languages as diverse as Python, Java, and Go can be aligned under a shared schema. We release the dataset publicly on Hugging Face and the accompanying codebase on GitHub, which includes complete pipelines for dataset reproduction, grammar compilation, and a visualization tool for exploring the unified AST across languages. Together, these resources establish MLCPD as an open, reproducible foundation for future research in cross-language representation learning and program analysis.

Problem

• Unifying syntactic code representations across ten programming languages
• Providing hierarchical tree structures with universal AST schema
• Enabling consistent cross-language reasoning and structural learning

Innovation

• Universal AST schema unifies ten programming languages
• Dataset provides hierarchical tree representations with metadata
• Enables cross-language structural learning and analysis