Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

2026-07-23

AI Summary

This study addresses the challenge of applying AI-assisted programming to data preparation in longitudinal research when personal data cannot be uploaded to the cloud due to governance constraints. The authors propose the first localized evaluation framework tailored to real-world scientific tasks, leveraging open-source large language models and an AI agent architecture to execute end-to-end R code generation—including category alignment and multi-wave data merging—on consumer-grade hardware. An automated mechanism evaluates both code correctness and output quality. Experiments demonstrate that state-of-the-art open-source models with 31–35B parameters achieve an average task completion rate of 87.9% across 20 tasks involving 102 variables, confirming the feasibility and practical potential of cloud-independent local AI agents for sensitive data processing. The work also releases real-world data-cleaning scripts and a benchmark for evaluation.

Abstract

Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises: a curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study), task definitions encompassing tasks such as category harmonization and multi-wave merging, and automated routines for evaluating the LLM-produced R code and outputted data. We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ("average task completion" up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: https://github.com/UCL-ARC/RRBench.

Problem

• longitudinal data preparation
• data governance
• open-weight LLMs
• privacy-preserving AI
• local deployment

Innovation

• open-weight LLMs
• local deployment
• data preparation
• longitudinal studies
• AI agent evaluation