Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning

2026-08-04

AI Summary

This work addresses the challenge that vision-language model (VLM)-based reward signals in reinforcement learning are often too noisy and unreliable for direct policy training. To overcome this limitation, the authors propose Structure-Aware Fine-Tuning (SAFT), a novel approach that incorporates the intrinsic task structure as an inductive bias into VLM reward modeling. SAFT employs self-supervised learning to refine reward signals online and applies LoRA adapters to impose structure-aware regularization in the VLM’s latent space—without requiring ground-truth supervision or human preference annotations. Experiments demonstrate that SAFT consistently denoises reward signals across multiple foundation models, significantly improving their alignment with ideal rewards as measured by EPIC distance, while accelerating policy convergence and enhancing training stability.

Abstract

Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass manual reward engineering. Although promising, these rewards are often noisy and unreliable, limiting their direct utility during deployment. We present Structure-Aware Fine-Tuning (SAFT), a simple, self-supervised method that refines these imperfect reward signals online without access to ground-truth supervision. SAFT leverages intrinsic structural priors to regularize the VLM's latent space via LoRA adapters. We rigorously evaluate SAFT across a spectrum of base model capabilities to demonstrate its versatility. Our results show that SAFT consistently denoises the reward landscape, yielding faster policy convergence and substantially improved alignment (EPIC distance) relative to the underlying base model, suggesting that failures can often be attributed to structural brittleness rather than semantic misunderstanding. By replacing extensive human preference annotation with structural inductive biases inherent to the task, SAFT offers a scalable path for stabilizing text-conditioned RL and underscores the broader value of incorporating task structure as a general inductive bias.

Problem

• reward models
• Vision-Language Models
• reinforcement learning
• noisy rewards
• reward engineering

Innovation

• Structure-Aware Fine-Tuning
• Vision-Language Models
• Reward Denoising
• LoRA Adapters
• Inductive Bias