in-context learning evaluation

This work investigates whether large language models (LLMs) rely on memorization of training data—rather than generalization—during in-context learning (ICL). Method: We propose a novel empirical memory probing framework, integrating cross-example-scale (zero-/few-/many-shot) correlation modeling with controlled prompt ablation experiments. Contribution/Results: Our study provides the first direct evidence that ICL significantly activates memorized training data. We find that unlabeled examples most readily trigger memory retrieval, and identify ~40% memory activation as a critical threshold for few-shot performance gains. Crucially, ICL performance improvement exhibits strong positive correlation with surfaced memory—especially beyond zero-shot settings—establishing “memory” as a core, previously underappreciated factor governing ICL effectiveness and challenging purely generalization-based explanatory paradigms.

This work investigates the underlying mechanism behind the strong correlation between the number of in-context examples and prediction performance in in-context learning (ICL). We propose the first Bayesian-theoretic ICL scaling law, modeling ICL as approximate Bayesian inference and explicitly characterizing the quantitative interplay among task prior, learning efficiency, and per-example information contribution. We validate the theory through extensive experiments: (i) empirical evaluation on GPT-2 variants; (ii) controlled synthetic-data studies; (iii) comparisons with supervised fine-tuning (SFT); and (iv) multi-example jailbreaking tests. Our law significantly outperforms existing scaling models in predictive accuracy; precisely forecasts the critical number of examples required for ICL-induced performance recovery post-SFT; and reveals an inherent limitation of post-training alignment—namely, its inability to fully suppress unsafe behaviors under multi-example prompting. Collectively, this work establishes a unified theoretical framework for interpreting ICL behavior, assessing safety risks, and analyzing alignment mechanisms.

This work addresses the limitations of existing evaluation methods for assessing large language models’ ability to use external tools in complex real-world scenarios, which often suffer from oversimplified toolsets, rigid workflows, or subjective scoring. To this end, we present the first large-scale benchmark grounded in real Model Context Protocol (MCP) servers, encompassing 36 MCP services, 220 tools, and 1,000 multi-step natural language tasks that require agents to autonomously discover and orchestrate multiple tools. The evaluation employs a no-tool-name prompting strategy and a fine-grained, fact-based scoring mechanism, supported by a containerized framework and multidimensional diagnostic metrics—including tool discovery, parameterization, and error recovery. Experiments reveal that state-of-the-art models achieve pass rates exceeding 50%, with primary failure modes stemming from insufficient tool utilization and task comprehension errors. The benchmark framework, task schema, and a public subset of 500 tasks are openly released.

This work addresses the proliferation of large language model (LLM) evaluation benchmarks, which has outpaced systematic assessment of their intrinsic quality. To this end, we propose Benchmark², a novel framework that establishes the first quantitative methodology for evaluating the reliability and validity of LLM benchmarks through three complementary metrics: cross-benchmark ranking consistency, discriminability score, and capability alignment bias. Empirical evaluation across 15 benchmarks and 11 LLMs demonstrates that Benchmark² not only reveals substantial quality disparities among existing benchmarks but also enables the construction of streamlined test sets that maintain high evaluative performance while significantly reducing assessment scale.

This work investigates the mechanistic roles and differential modality contributions of image and text examples in multimodal in-context learning (ICL). Addressing the fundamental question—“Why is multimodal ICL effective?”—we propose an empirical perturbation-based analytical framework, which, for the first time, uncovers modality-specific effects and model-induced inductive biases. We design a task-adaptive, modality-driven example construction strategy, rigorously validated through controlled modality perturbations, cross-scale model evaluation, and semantic consistency checking. Experiments span diverse multimodal tasks, yielding substantial ICL performance gains. Our findings yield a transferable, task-aware demonstration design guideline, advancing both the theoretical understanding and practical methodology of multimodal prompt engineering.

This work addresses the limited ability of current language models to dynamically acquire and apply new knowledge—such as domain-specific rules or empirical laws—from task contexts in complex real-world scenarios. To systematically define and evaluate this contextual learning capability, the authors introduce CL-bench, a novel benchmark comprising 500 expert-designed complex contexts, 1,899 tasks, and 31,607 validation rules, structured as context-task-validation triplets. Evaluation across ten state-of-the-art models reveals a significant performance bottleneck, with models completing only 17.2% of tasks on average; even the best-performing model, GPT-5.1, achieves just 23.7%. This benchmark fills a critical gap in assessing dynamic knowledge acquisition and application, highlighting a key limitation in contemporary language models’ contextual learning capacities.

This work addresses the lack of a unified modeling framework and rigorous theoretical analysis for in-context learning (ICL). It proposes the first general probabilistic modeling framework grounded in statistical inference and exponential family distribution theory. Within this framework, the authors systematically derive performance bounds for ICL under both general and exponential-family parameter distributions. They further provide a quantitative analysis of how key factors—such as the number of demonstration examples, model parameter sensitivity, and similarity between demonstrations and query inputs—affect ICL effectiveness. By identifying the critical determinants of ICL performance, this study establishes an interpretable theoretical foundation for understanding the contextual learning mechanisms of large language models.

This work proposes a purely reinforcement learning (RL)–driven training framework for large language models to overcome their limitations in complex tasks stemming from reliance on internal knowledge and ineffective external tool invocation. Departing from conventional supervised fine-tuning (SFT), the approach introduces a small number of in-context examples during rollouts to guide tool usage and employs a progressive example annealing strategy, ultimately enabling zero-shot tool utilization. By integrating contextual reinforcement learning with standard RL algorithms, the method achieves state-of-the-art performance across multiple reasoning and tool-use benchmarks. Notably, it demonstrates for the first time that highly effective and scalable tool-calling capabilities can be trained without any supervised fine-tuning.

This work addresses the limitation of existing video understanding benchmarks, which primarily assess static knowledge and fail to evaluate a model’s ability to acquire procedural knowledge from few dynamic demonstrations. To this end, we introduce Demo-ICL, the first in-context learning task tailored for procedural videos, along with Demo-ICL-Bench—a novel benchmark supporting multimodal (text and video) demonstrations. We further propose a two-stage training strategy that combines video-supervised fine-tuning with information-augmented direct preference optimization to enhance the capacity of multimodal large language models to generalize procedural knowledge from demonstrations. Experimental results show that current models perform poorly on this task, whereas our approach significantly improves in-context learning performance, opening a new direction for video understanding research.

This work addresses the limitations of existing representation engineering approaches, which rely on synthetic data and suffer from irreproducible evaluations and susceptibility to superficial patterns. The authors construct the first large-scale, multi-source aligned capability representation framework grounded in real-world benchmarks, curated from over 10,000 academic papers and hundreds of public datasets, spanning 94 distinct capabilities. This framework enables cross-benchmark aggregation of capability vectors and transferable evaluation, effectively mitigating bias from any single data source. Experiments across 12 large language models reveal that benchmark-pooled capability vectors exhibit stable clustering structures; differential mean achieves the best performance in 10 models, while logistic regression outperforms others across the greatest number of capability–model combinations, underscoring the critical influence of both evaluation dimensions and readout methodologies.