Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering

2026-07-28

AI Summary

This work addresses the security risks posed by widely reused, third-party visual language model (VLM) components in the AI supply chain, which constitute critical trust boundaries vulnerable to backdoor attacks. The paper introduces the first architecture-based, representation-guided backdoor mechanism that embeds trigger-controlled additive perturbations into intermediate representations—without contaminating training data or altering input prompts—to enable stealthy manipulation of downstream behaviors. This approach requires no retraining and successfully induces integrity violations, safety bypasses, and ranking biases across diverse tasks such as visual question answering, text-to-image generation, and retrieval, while preserving normal performance on clean inputs. The findings highlight the previously underappreciated risk that shared model artifacts can carry concealed, malicious logic.

Abstract

Vision--Language Models (VLMs) are increasingly deployed through a model supply chain in which pretrained checkpoints, architecture definitions, text encoders, and exported computation graphs are distributed by third parties and reused across downstream services. This reuse model creates a security-critical trust boundary: VLM deployments inherit not only learned parameters but also executable behavior encoded in shared model artifacts. In this paper, we show that a malicious provider can exploit this trust boundary by embedding architectural backdoors into VLM supply chains through representation steering. Our attack introduces dormant steering logic into the model architecture through a trigger-gated additive modification of an intermediate representation, without poisoning training data, controlling downstream fine-tuning, or modifying prompts at deployment time. When the trigger is absent, the modification reduces to zero and the model follows its normal computation, preserving clean utility. When the trigger is present, a steering direction shifts the internal representation toward an attacker-defined objective. We evaluate the attack across multiple VLM families and downstream tasks, including visual question answering, text-to-image generation, retrieval, and semantic response biasing. The results show that the proposed architectural steering backdoor compromises integrity, safety enforcement, and ranking fairness while preserving normal behavior on clean inputs. We further show that shared VLM artifacts can carry dormant steering logic against downstream services, and we propose an auditing defense that inspects the executable logic distributed with model artifacts rather than only their learned weights.

Problem

• architectural backdoors
• vision-language models
• model supply chain
• representation steering
• security

Innovation

• architectural backdoors
• representation steering
• vision-language models
• model supply chain
• trigger-gated modification