embedding paradigm comparison

This work addresses the gap between benchmark-driven embedding model selection and real-world deployment constraints by introducing the first framework to evaluate embedding models within a complete retrieval pipeline. It systematically compares the end-to-end performance of T3EM’s commercial API against leading open-source models across diverse tasks—including retrieval, classification, clustering, and semantic similarity—as covered by the MTEB benchmark, while jointly accounting for latency, cost, task type, and deployment conditions. The study develops a comprehensive, end-to-end model selection guide encompassing embedding generation, indexing, search, and chunking strategies, revealing significant performance discrepancies that emerge only in full-system contexts. These insights provide practitioners with actionable, empirically grounded criteria for embedding model adoption in real-world applications.

This work addresses the absence of standardized benchmarks for end-to-end performance and quality evaluation in retrieval-augmented generation (RAG) systems. The authors propose a modular and configurable RAG benchmarking framework that, for the first time, enables decoupled assessment of individual pipeline stages—including embedding, indexing, retrieval, reranking, and generation. The framework supports multimodal data, multiple vector databases (e.g., Milvus, Qdrant), and diverse large language models, while accommodating realistic query loads and update patterns. It automatically collects key metrics such as throughput, resource utilization, and accuracy. Experimental results demonstrate that the framework introduces negligible performance overhead while providing comprehensive evaluation of RAG system effectiveness. The implementation is publicly released as open-source software.

Current language model benchmarks often suffer from coarse-grained metadata, making it difficult to accurately assess their coverage of capabilities that matter to users. To address this limitation, this work proposes a fine-grained retrieval system based on natural language queries that precisely identifies evaluation items relevant to real-world usage scenarios across 20 mainstream benchmarks. For the first time, the system leverages interpretable retrieval evidence to expose gaps between benchmark content and user intent. It further enables transparent validation of benchmark validity through human evaluation combined with analyses of content validity and construct validity. Human assessment confirms that the method achieves high retrieval precision and effectively uncovers issues such as insufficient capability coverage or unstable scoring.

This work addresses the proliferation of large language model (LLM) evaluation benchmarks, which has outpaced systematic assessment of their intrinsic quality. To this end, we propose Benchmark², a novel framework that establishes the first quantitative methodology for evaluating the reliability and validity of LLM benchmarks through three complementary metrics: cross-benchmark ranking consistency, discriminability score, and capability alignment bias. Empirical evaluation across 15 benchmarks and 11 LLMs demonstrates that Benchmark² not only reveals substantial quality disparities among existing benchmarks but also enables the construction of streamlined test sets that maintain high evaluative performance while significantly reducing assessment scale.

Prior work on generative tasks (e.g., text-to-SQL, question answering) largely overlooks table retrieval—a critical prerequisite for leveraging structured data. Method: We introduce TARGET, the first table-level retrieval benchmark explicitly designed for generative tasks, featuring a structured evaluation framework with multi-source real-world tabular datasets, fine-grained human annotations, and an integrated retrieval-generation evaluation pipeline. Contribution/Results: Experiments show dense retrieval using BERT-based table encoders substantially outperforms BM25 (mAP gain >40%). Metadata absence—especially table titles—degrades performance by up to 35%. Significant performance disparities exist across datasets and tasks. Effective table retrieval boosts SQL generation accuracy by up to 22%. This work is the first to systematically quantify the impact of table retrieval on downstream generative performance and establishes a standardized evaluation paradigm for structured-data retrieval.

This work addresses the limitations of existing representation engineering approaches, which rely on synthetic data and suffer from irreproducible evaluations and susceptibility to superficial patterns. The authors construct the first large-scale, multi-source aligned capability representation framework grounded in real-world benchmarks, curated from over 10,000 academic papers and hundreds of public datasets, spanning 94 distinct capabilities. This framework enables cross-benchmark aggregation of capability vectors and transferable evaluation, effectively mitigating bias from any single data source. Experiments across 12 large language models reveal that benchmark-pooled capability vectors exhibit stable clustering structures; differential mean achieves the best performance in 10 models, while logistic regression outperforms others across the greatest number of capability–model combinations, underscoring the critical influence of both evaluation dimensions and readout methodologies.

This work addresses the semantic gap between evaluation metrics and training data in large model pretraining, which hinders precise diagnosis and remediation of capability deficiencies. The authors propose “capability slices” as fundamental units aligning evaluation and data, establishing a bidirectional classification framework that links evaluation tasks with non-instructional training data through explicit mapping rules. This enables a closed-loop pipeline from evaluation failures to targeted data interventions. For the first time, the approach supports auditable and systematic reasoning that translates evaluation signals into data corrections, moving beyond intuition-driven tuning paradigms. Experiments demonstrate its efficacy in both directions: repairing specific training loss components restores BBH performance to 66.44, while targeted data sampling boosts AIME2025/2026 Pass@128 from 6.67/0.00 to 26.67.

This work addresses the limitations of existing RAG evaluation frameworks in identifying critical failure modes in enterprise-grade multi-turn dialogues—such as case misidentification, workflow misalignment, and partial resolution across turns. To this end, we propose a case-aware evaluation framework that, for the first time, incorporates case-workflow alignment as a core evaluation dimension. The framework introduces eight operation-oriented metrics for fine-grained analysis of each conversational turn and employs a severity-aware scoring mechanism to mitigate score inflation and enhance diagnostic precision. Built upon an LLM-as-a-Judge architecture with deterministic prompting and strict JSON output formatting, our approach enables interpretable, production-ready, and batch-deployable evaluation. Experimental results demonstrate that the framework effectively uncovers key performance trade-offs in enterprise settings that are invisible to conventional agent-level metrics, thereby delivering actionable insights for system optimization.

This work addresses the limitations of traditional static benchmarks—prone to saturation, contamination, and high updating costs—and the susceptibility of existing large language model (LLM) auto-scoring methods to prompt sensitivity and bias. It proposes the first three-stage framework that evaluates LLMs’ *benchmark design capability* rather than merely their question-answering performance. The approach leverages structured domain cards for extraction, quota-based multi-model collaborative item generation, and scoring via precise, numerical, and symbolic verifiers combined with psychometric analysis. From nine domains, it generates 16.7K items (retaining 15K core items) and constructs a designer–responder matrix with 152K scoring records. Empirical results reveal only a moderate correlation between design and answering abilities (Spearman ρ ≈ 0.37) and a strong negative association between invalid items and discrimination (r ≈ −0.62), demonstrating the framework’s effectiveness for scalable, cross-modal, and multilingual benchmark auditing.