Whitespaces Don't Lie: Feature-Driven and Embedding-Based Approaches for Detecting Machine-Generated Code

2026-01-27

AI Summary

This work addresses the emerging challenges posed by large language models generating code—particularly concerning academic integrity, authorship attribution, and responsible AI use—by proposing and systematically comparing two detection approaches. The first leverages lightweight, interpretable surface-level features of code style and structure, such as whitespace and indentation patterns, while the second employs semantic representations derived from pretrained code embeddings like CodeBERT. Experimental results demonstrate that the feature-engineering approach slightly outperforms the embedding-based method, achieving a ROC-AUC of 0.995 and an F1 score of 0.971 compared to 0.994 and 0.965, respectively. These findings highlight the strong discriminative power of superficial syntactic cues and elucidate the trade-off between interpretability and generalization, offering complementary and effective technical pathways for identifying the provenance of AI-generated code.

Technology Category

Natural Language Processing: Code Generation / Program Synthesis from Natural LanguageMachine Learning: Large Multimodal Models (LMMs)Computer Vision: Large Vision Models

Application Category

Search and Retrieval-Augmented AI: Web evaluation methodologies and metricsWeb Mining and Content Analysis: Large pretrained models with web dataGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) have made it remarkably easy to synthesize plausible source code from natural language prompts. While this accelerates software development and supports learning, it also raises new risks for academic integrity, authorship attribution, and responsible AI use. This paper investigates the problem of distinguishing human-written from machine-generated code by comparing two complementary approaches: feature-based detectors built from lightweight, interpretable stylometric and structural properties of code, and embedding-based detectors leveraging pretrained code encoders. Using a recent large-scale benchmark dataset of 600k human-written and AI-generated code samples, we find that feature-based models achieve strong performance (ROC-AUC 0.995, PR-AUC 0.995, F1 0.971), while embedding-based models with CodeBERT embeddings are also very competitive (ROC-AUC 0.994, PR-AUC 0.994, F1 0.965). Analysis shows that features tied to indentation and whitespace provide particularly discriminative cues, whereas embeddings capture deeper semantic patterns and yield slightly higher precision. These findings underscore the trade-offs between interpretability and generalization, offering practical guidance for deploying robust code-origin detection in academic and industrial contexts.

Problem

• machine-generated code
• code detection
• authorship attribution
• academic integrity
• code origin

Innovation

• whitespace features
• code origin detection
• feature-based detection
• embedding-based detection
• CodeBERT