1GC-7RC: One Graphic Card -- Seven Research Challenges! How Good Are AI Agents at Doing Your Job?

Unknown Date

AI Summary

This work addresses the absence of standardized benchmarks for evaluating AI agents’ ability to autonomously design, implement, and train machine learning models from scratch across diverse tasks. To this end, we introduce the first modular and extensible single-GPU multitask benchmark that strictly constrains agents to modify only the training code while operating under fixed time and computational budgets across seven distinct tasks. By fixing data preprocessing and evaluation pipelines, providing a unified training template, and integrating a range of open- and closed-source coding agents—including GPT, Claude Code, Qwen, and Kimi—we systematically assess 35 agent–task combinations. Our experiments reveal substantial differences among agents in their implicit machine learning knowledge, planning capabilities, and time management. All code and evaluation data are publicly released.

Abstract

Autonomous AI coding agents are becoming a core tool for ML practitioners in industry and research alike. Despite this growing adoption, no standardized benchmark exists to evaluate their ability to design, implement, and train models from scratch across diverse domains. We introduce **1GC-7RC** (*Single Graphic Card: Seven Research Challenges*), a benchmark comprising seven ML tasks spanning language modeling, image classification, semantic segmentation, graph learning, tabular prediction, time-series forecasting, and text classification. Each task provides a locked data-preparation and evaluation script together with a baseline training script; the agent may only modify the training code, has no access to pretrained weights (with one controlled exception for semantic segmentation), no internet access, and must complete each task within a task-specific wall-clock budget (40-120 minutes) on a single GPU. We evaluate seven coding agents: five proprietary (Claude Code with Sonnet 4.6, Opus 4.6, and Opus 4.7; Codex CLI with GPT 5.5; and OpenCode with Qwen 3.6+) and two open-source (OpenCode with Kimi K2.5, Kimi K2.6). Across 5 runs per agent-task pair, we report substantial performance differences that reveal varying levels of implicit ML knowledge, planning ability, and time-budget management. The benchmark, harness, and all evaluation artifacts are publicly available on GitHub at https://github.com/Strolchii/1GC-7RC-Benchmark to facilitate reproducible comparison of future agents. Because our benchmark design is modular, the benchmark can be extended to new tasks and domains, adapted to different GPU budgets, and used to study multi-agent settings, making it a flexible platform for future research on autonomous research agents.

Problem

• AI coding agents
• benchmark
• machine learning
• autonomous research
• model training

Innovation

• autonomous AI agents
• standardized benchmark
• single-GPU constraint
• modular evaluation framework
• code generation for ML