VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models

2025-09-29

AI Summary

Existing behavioral control methods for large vision-language models (VLMs) require internal model access, are vulnerable to instruction-level overrides, and lack compatibility with API-based or closed-source deployments. To address these limitations, this paper proposes VISOR++, a parameter-free, runtime-agnostic visual input optimization framework for universal behavioral steering. VISOR++ employs a single learnable image to induce cross-model consistent behavioral priors—such as refusal, sycophancy, or survival instincts—across diverse VLMs. Its core innovations include activation-matching-driven visual optimization, multi-model ensemble training, and behavior-direction alignment. Evaluated on LLaVA and IDEFICS2, VISOR++ matches the performance of conventional steering vectors while demonstrating strong generalization to unseen models. Crucially, it preserves 99.9% of original accuracy on a 14,000-sample MMLU benchmark, confirming minimal task performance degradation.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As Vision Language Models (VLMs) are deployed across safety-critical applications, understanding and controlling their behavioral patterns has become increasingly important. Existing behavioral control methods face significant limitations: system prompting approaches could easily be overridden by user instructions, while applying activation-based steering vectors requires invasive runtime access to model internals, precluding deployment with API-based services and closed-source models. Finding steering methods that transfer across multiple VLMs is still an open area of research. To this end, we introduce universal visual input based steering for output redirection (VISOR++), to achieve behavioral control through optimized visual inputs alone. We demonstrate that a single VISOR++ image can be generated for an ensemble of VLMs to emulate each of their steering vectors. By crafting universal visual inputs that induce target activation patterns, VISOR++ eliminates the need for runtime model access while remaining deployment-agnostic. This means that when an underlying model supports multimodal capability, model behaviors can be steered by inserting an image input replacing runtime steering vector based interventions. We first demonstrate the effectiveness of the VISOR++ images on open-access models such as LLaVA-1.5-7B and IDEFICS2-8B along three alignment directions: refusal, sycophancy and survival instinct. Both the model-specific steering images and the jointly optimized images achieve performance parity closely following that of steering vectors for both positive and negative steering tasks. We also show the promise of VISOR++ images in achieving directional behavioral shifts for unseen models including both open-access and closed-access ones. Furthermore, VISOR++ images are able to preserve 99.9% performance on 14,000 unrelated MMLU evaluation tasks.

Problem

• Achieving behavioral control in Vision Language Models without runtime access
• Developing universal visual inputs to steer multiple VLMs simultaneously
• Overcoming limitations of system prompts and invasive activation interventions

Innovation

• Universal visual inputs steer model behaviors
• Optimized images replace runtime activation interventions
• Single image works across multiple vision language models