FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance

2026-02-02

AI Summary

This work addresses the opacity of internal feature dependency mechanisms in multimodal foundation models and the challenge of actively controlling these mechanisms without altering task semantics. The authors propose an instruction-conditioned, parameter-efficient adaptation framework that, for the first time, treats natural language instructions as computational control signals. By integrating feature-group-aligned LoRA modules with an instruction-driven gating mechanism, the method enables selective enhancement or suppression of core versus spurious feature groups. Notably, this approach operates without modifying the label space or training objectives. Experiments on image–text and audio–video benchmarks demonstrate that the framework can causally steer internal model computations, substantially improving robustness against spurious feature interference.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal foundation models integrate heterogeneous signals across modalities, yet it remains poorly understood how their predictions depend on specific internal feature groups and whether such reliance can be deliberately controlled. Existing studies of shortcut and spurious behavior largely rely on post hoc analyses or feature removal, offering limited insight into whether reliance can be modulated without altering task semantics. We introduce FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework that enables explicit control over internal feature reliance while keeping the predictive objective fixed. FiLoRA decomposes adaptation into feature group-aligned LoRA modules and applies instruction-conditioned gating, allowing natural language instructions to act as computation-level control signals rather than task redefinitions. Across text--image and audio--visual benchmarks, we show that instruction-conditioned gating induces consistent and causal shifts in internal computation, selectively amplifying or suppressing core and spurious feature groups without modifying the label space or training objective. Further analyses demonstrate that FiLoRA yields improved robustness under spurious feature interventions, revealing a principled mechanism to regulate reliance beyond correlation-driven learning.

Problem

• multimodal foundation models
• feature reliance
• spurious correlations
• controllable adaptation
• internal feature groups

Innovation

• FiLoRA
• instruction-conditioned gating
• feature reliance control
• parameter-efficient adaptation
• multimodal foundation models