Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

2026-10-08

AI Summary

This study addresses the challenge that small vision-language models (VLMs) struggle to effectively retrieve external evidence and require optimized task delegation with external frameworks. To this end, it proposes an offline compilation framework in which a large teacher model dynamically adjusts reusable content and control logic based on execution traces, allocates high- and low-cost operational decisions according to empirical behavior, and supports interface recompilation for novel models. Through offline program synthesis and weight-free deployment, the proposed framework yields improvements of 9.9–23.9 points across seven VQA tasks. Notably, it outperforms LoRA fine-tuning in few-shot scenarios, substantially enhancing the external tool-calling capabilities of small models without requiring parameter updates.

Abstract

Small vision-language models may be able to read external evidence yet struggle to obtain it. We introduce Harness Compilation (HC), an offline procedure that adapts the division of work between a frozen small VLM and its external harness. A large teacher uses student execution traces to revise reusable content and control, while a separate validation set selects the deployed harness. Deployment requires neither weight updates nor teacher calls. Across seven visual question-answering settings with students of at most 9B parameters, HC improves scores over bare students by 9.9-23.9 points, averaged over three independent builds per setting. Interventions on five runtime decision types (invocation, selection, argument generation, evidence integration and abstention) show why this allocation matters: requesting evidence and generating open queries can be costly, whereas bounded choices and reading supplied text can remain useful student work. Fact cards benefit all ten evaluated students, but decision policies transfer unevenly. Recompilation for a new student model helps when the transferred interface no longer fits the student. With 100 practice items, HC exceeds answer-only LoRA on three tasks. Larger training budgets can match or surpass a fixed harness, while combining the two improves SlideVQA beyond either alone. These findings support allocating work from measured student behavior rather than uniformly removing decisions.

Problem

• vision-language models
• task allocation
• external harness
• knowledge distillation
• visual question answering

Innovation

• Harness Compilation
• Vision-Language Model
• Knowledge Distillation
• Decision Allocation
• Tool Use