Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training

2026-06-10

AI Summary

This work addresses the deployment challenges of existing parameter-efficient fine-tuning methods—such as LoRA and Soft Prompting—which require modifications to the model’s computation graph and are thus incompatible with high-throughput inference engines like vLLM. The authors propose ART, a novel approach that treats visual inputs as learnable “computational art.” By optimizing only pixel-level visual prompts while keeping all parameters of the multimodal large language model frozen, ART injects task-specific information without altering the model architecture or computation graph. Consequently, it seamlessly integrates with precompiled inference systems and supports arbitrary fine-tuning objectives. Experimental results demonstrate that ART achieves performance on par with LoRA on Qwen-series models across multiple textual benchmarks, particularly excelling in mathematical reasoning and structured tool-use tasks.

Abstract

There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use benchmarks.

Problem

• Parameter-Efficient Fine-Tuning
• Multi-modal LLMs
• Computational Graph Modification
• High-throughput Inference
• Precompiled Models

Innovation

• ART
• Parameter-Efficient Fine-Tuning
• Multimodal LLM
• Visual Prompting
• Frozen Model