Benchmarking LLMs on File System Design and Implementation: The Good, The Bad, and The Ugly

2026-07-31

AI Summary

This work addresses the absence of a dedicated evaluation framework for assessing large language models (LLMs) on file system–related tasks. To bridge this gap, we introduce φ-Bench, the first comprehensive benchmark specifically designed for evaluating LLMs across six core file system task categories: comprehension, implementation, modeling, debugging, optimization, and novel feature development. Leveraging expert knowledge, textbook content, and AI-assisted generation techniques, we construct a high-quality dataset comprising 505 diverse tasks. We conduct systematic evaluations of prominent open-source and closed-source LLMs, revealing significant performance disparities and characteristic failure modes across task types. Based on our empirical analysis, we propose targeted mitigation strategies. The φ-Bench dataset will be publicly released to foster further research in this domain.

Abstract

Large Language Models (LLMs) are fundamentally transforming computer system research and development. As we employ LLMs in file system (fs) development, it is essential to understand their capabilities, limitations, and operational efficiency for domain-specific tasks. We present φ-Bench, an LLM benchmarking framework for fs-specific tasks. To facilitate benchmarking, we develop six types of tasks in φ-Bench: basic understanding, basic implementation, performance modeling, debugging, optimization, and new feature development. Each type emphasizes different LLM capabilities: instruction following, knowledge recall, reasoning, or coding. To create high-quality tasks while achieving broad coverage with minimal human effort, we develop a new AI-assisted task generation pipeline in addition to expert-written and textbook-adapted tasks. With 505 tasks in φ-Bench, we conduct an empirical study with both open source (DeepSeek-V4-Flash, GLM-5.1, and MiniMax-M2.7) and proprietary (Claude-Opus-4.7, GPT-5.2, and Gemini-3.1-Pro) LLMs. Our study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures. We will open source φ-Bench to facilitate public research on using LLMs for fs development.

Problem

• Large Language Models
• File System
• Benchmarking
• System Development
• Domain-specific Tasks

Innovation

• LLM benchmarking
• file system development
• AI-assisted task generation
• system software evaluation
• empirical study