Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

2026-06-03

AI Summary

This work addresses the limitations of multimodal in-context learning (ICL), which suffers from high inference latency and sensitivity to the formatting and ordering of demonstration examples, leading to poor stability. To overcome these issues, the authors propose Hyper-ICL, a framework that reconstructs the effect of demonstrations without requiring explicit example inputs at inference time. Hyper-ICL employs lightweight low-rank adapters augmented with a query-adaptive modulation mechanism that enables dynamic, layer-wise and head-wise intervention. Furthermore, it introduces an anchor distillation loss based on geodesic distances in hyperbolic space to effectively model the relationship between queries and demonstrations. Experiments demonstrate that Hyper-ICL significantly improves both accuracy and robustness across six major multimodal benchmarks—including VQAv2, OK-VQA, and COCO Caption—outperforming standard ICL and current state-of-the-art approaches.

Abstract

Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.

Problem

• Multimodal In-Context Learning
• inference latency
• instability
• demonstration sensitivity
• attention calibration

Innovation

• Hyperbolic Anchor Distillation
• Attention Calibration
• Demonstration-Free ICL
• Query-Adaptive Modulation
• Multimodal In-Context Learning