AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

2026-09-28

AI Summary

This study addresses the challenge of efficiently adapting multimodal large models to downstream tasks under limited supervision, particularly when parameter fine-tuning is constrained. To this end, we propose AdaKerNet, a neural kernel decoder that operates without access to model parameters. The method integrates Lipschitz-constrained feature extraction, reference kernels, and soft structural prior modeling, jointly learning features and structures via a lightweight nonlinear predictor to enable adaptive prediction over frozen representations. Experimental results demonstrate that AdaKerNet significantly outperforms baseline methods across four multimodal large models and diverse input modalities, reducing average error by up to 41%. These findings confirm its effectiveness in enhancing adaptation performance under few-shot scenarios.

Abstract

Large foundation models have been introduced with the promise of efficient adaptation to downstream tasks. Yet, under limited supervision, MLLMs, an important class of large foundation models, remain challenging to adapt to various downstream tasks. Adaptation typically relies either on MLLM parameter fine-tuning or on training neural-based decoders. Both approaches struggle under limited supervision, while fine-tuning additionally requires access to model parameters, which is often unavailable for closed-source models. We introduce AdaKerNet, a novel learnable task-adaptive neural kernel decoder. AdaKerNet is fully agnostic to the parameters of the underlying MLLM and operates solely on its (frozen) rich representations obtained from the diverse available modalities. AdaKerNet relies on (i) a set of learnable, Lipschitz-controlled multimodal features derived from these MLLM representations; (ii) a reference kernel that provides a soft structural prior on those features; and (iii) a lightweight nonlinear neural predictor that adaptively deforms that structure. Learning the kernel representation and the neural predictor jointly within a unified optimization framework allows AdaKerNet to capture features and geometric relationships relevant to the downstream task. Numerical tests across four MLLMs: BLIP-2, LLaVA-1.5, Qwen2.5-VL, and Gemini Embedding 2, and multimodal inputs spanning text, audio, images, and tabular measurements demonstrate significant and consistent improvements over direct MLP, attention-, autoencoder- and kernel-based decoders, across a range of scarce-label budgets, with average error reduction of up to 41% across baselines. These results establish AdaKerNet as an effective approach for prediction from frozen multimodal representations in the scarce label regime. Additional structural ablations highlight the complementary contributions of AdaKerNet's components.

Problem

• Multimodal Large Language Models
• Limited Supervision
• Task Adaptation
• Frozen Representations
• Downstream Prediction

Innovation

• Neural Kernel Decoding
• Task-Adaptive Prediction
• Multimodal Large Models
• Few-Shot Learning
• Parameter-Free Adaptation