"Skill issues'': data-centric optimization of lakehouse agents

2026-05-31

AI Summary

This work addresses the challenge that agent task execution in data lakehouse environments heavily relies on skill scripts and environmental configurations. To overcome this limitation, the authors propose an optimization approach based on task-validator pairs. The method executes candidate skills within isolated sandboxes and shifts the evaluation paradigm from output matching to state validation by combining trajectory signals with programmatic checks of the lakehouse state. This framework leverages a branchable lakehouse (Bauplan) architecture, Headless APIs, Git-like data primitives, and write-path data workflows to enable verifiable skill optimization. Preliminary experiments across 25 tasks demonstrate that the optimized skills achieve a 31.9% improvement in task accuracy.

Abstract

Coding agents are becoming users of data infrastructure, but their success depends not only on model quality: it also depends on the skills and environment files that teach agents how to use a system. We study how to optimize these artifacts for agents operating on a branching lakehouse, Bauplan. In our setting, headless APIs and Git-like data primitives expose data workflows through code, branches, commits, and merges. Our central observation is that a branching lakehouse turns data-agent evaluation from an output-matching problem into a state-verification problem: agent-generated pipeline code induces concrete, inspectable lakehouse changes. We present a data-centric optimization pipeline that generates task-verifier pairs, executes candidate skills in isolated sandboxes, and scores trajectories using both trace-level signals and programmatic checks over lakehouse state. In a preliminary evaluation on 25 tasks, optimized skills improve accuracy by 31.9%. These results suggest that write-path data workflows provide a useful substrate for optimizing agent skills beyond read-only tasks.

Problem

• lakehouse
• agent skills
• data-centric optimization
• state verification
• data workflows

Innovation

• lakehouse
• data-centric optimization
• agent skills
• state verification
• branching data workflows