downstream evaluation

Existing probabilistic forecasting evaluation metrics primarily emphasize predictive accuracy while neglecting their practical utility in downstream decision-making tasks, leading to a misalignment between evaluation and application. To address this, we propose a data-driven evaluation alignment framework that formulates the learning of a surrogate evaluation function as an end-to-end optimization problem. Leveraging proper scoring rule theory, our approach employs a neural network-parameterized weighted scoring rule to automatically learn an evaluation function aligned with downstream objectives—without assuming any prior cost structure. This work is the first to formalize evaluation alignment as a learnable problem, combining theoretical rigor with engineering scalability. Experiments on synthetic and real-world regression tasks demonstrate its effectiveness: it significantly reduces the gap between evaluation scores and downstream decision utility, enabling rapid, task-adaptive model selection and hyperparameter tuning.

Configuration space explosion complicates performance impact modeling, while gray-box approaches rely on structural knowledge (e.g., module execution graphs) to improve model accuracy—yet the mechanisms by which structural features (e.g., number of modules or configuration options) and structural knowledge influence modeling difficulty and optimization potential remain unclear. Method: We formally define “modeling hardness” and “improvement opportunity,” establishing an analytical framework and matrix to quantify the interplay among system structural complexity, structural knowledge level, and modeling benefit. Controlled experiments on synthetic systems integrate module execution graph analysis with gray-box modeling. Contribution/Results: We identify module count and configuration option count as dominant determinants of modeling hardness. Under high hardness, strong structural knowledge significantly increases improvement opportunity. Structural knowledge primarily enhances ranking accuracy, whereas hardness predominantly degrades prediction accuracy. Our findings provide theoretical foundations and strategic guidance for allocating structural knowledge investment according to specific modeling objectives.

This study addresses the challenge of deploying large language models in resource-constrained environments due to their high computational costs. To this end, the authors systematically evaluate the performance and efficiency of 16 language models, ranging from 0.5B to 3B parameters, across five categories of NLP tasks. They introduce a novel task-specific efficiency analysis framework and propose a Performance-Efficiency Ratio (PER) metric, which integrates accuracy, throughput, memory footprint, and latency through geometric mean normalization. Experimental results demonstrate that smaller models consistently achieve superior PER scores across all evaluated tasks, offering both quantitative justification and practical guidance for efficient inference deployment in real-world scenarios.

In scientific computing applications—such as trajectory prediction, optimal control, and minimum energy path computation—downstream algorithms critically depend on accurate model evaluations. Conventional mean-squared-error-based supervised learning often induces task-specific performance degradation due to misalignment between the loss function and the ultimate algorithmic objective. Method: We propose a task-oriented predictive modeling paradigm that replaces standard regression losses with a surrogate objective: the maximum prediction error over a downstream task support set. Our framework integrates sampling measure modeling, empirical risk discretization, and iterative optimization to directly optimize downstream algorithmic performance. Contribution/Results: This is the first approach to explicitly embed downstream robustness requirements into the training objective. Evaluated across multiple scientific computing benchmarks, it consistently improves both predictive accuracy and algorithmic stability, demonstrating superior generalization under task-relevant perturbations.

Communication barriers between data scientists and domain experts arise from oversimplified, accuracy-centric model performance reporting, hindering shared understanding of model limitations and contextual applicability. Method: We propose a visualization-mediated model explanation framework grounded in human-computer interaction principles, participatory design, and visual narrative techniques. This yields the first domain-expert-oriented model communication guideline—emphasizing risk, trade-offs, and situational appropriateness rather than isolated metrics like accuracy. An iterative empirical study was conducted using regression models, incorporating structured expert feedback for evaluation. Contribution/Results: The framework significantly improves domain experts’ ability to identify model limitations, recognize inherent trade-offs, and proactively make context-driven adoption decisions. Its core innovation lies in repositioning visualization as an interdisciplinary consensus-building medium—shifting the paradigm from “metric reporting” to “collaborative understanding.”

Current measures of AI reliance primarily rely on output adoption or subjective self-reports, which inadequately capture the allocation of cognitive effort between users and AI during task execution. This work proposes a counterfactual workflow-based simulation method that models the steps users would take without AI assistance to quantify the proportion of cognitive effort offloaded to the AI. Introducing a novel metric—the Offloading Score—it provides a more precise measure of AI dependence. This score effectively captures dynamic shifts in reliance under time pressure, facilitating both user self-reflection and system-level interventions. In a programming study with 40 developers, the Offloading Score detected a statistically significant 43% increase in reliance under time pressure (p = 0.018), outperforming conventional metrics and revealing that heightened dependence manifests as increased delegation of subtasks and direct reuse of AI-generated outputs.

Software systems face challenges in multi-objective performance modeling due to vast configuration spaces and low sampling efficiency. Method: This paper proposes LLM4Perf—the first large language model (LLM)-based feedback-driven collaborative sampling framework. It innovatively integrates semantic information from configuration documentation with runtime performance feedback to enable dynamic configuration space pruning and online optimization of sampling strategies. Contribution/Results: Unlike conventional approaches, LLM4Perf empirically demonstrates, for the first time, the LLM’s generalizable pruning capability in performance modeling—significantly enhancing multiple baseline methods. Across 112 evaluation scenarios, it achieves optimal performance in 68.8%; across 448 baseline experiments, 91.5% show performance improvement attributable to its pruning mechanism. This work establishes a reproducible framework and robust empirical foundation for LLM-enabled performance engineering.

This work addresses the misalignment between offline evaluation metrics and online performance objectives in industrial applications by establishing a unified theoretical framework that systematically quantifies the relationships among diverse evaluation metrics for the first time. By introducing the concepts of Bayes-optimal sets and regret transfer mechanisms, the study reveals structural asymmetries among metrics and provides a principled classification and relational modeling of metrics with varying mathematical forms. Theoretically characterizing metric consistency and transferability, this research offers novel insights and a methodological foundation for designing offline evaluation systems that are aligned with online objectives and backed by rigorous theoretical guarantees.

This study addresses the limited scope of traditional high-performance computing (HPC) evaluations, which typically focus solely on performance and energy consumption while overlooking the comprehensive environmental costs of operational configurations. The authors propose the first job-level unified accounting framework that integrates both operational and full life-cycle (embodied) carbon and water footprints. Leveraging life-cycle assessment methodologies, real-time runtime monitoring, and hardware manufacturing emission data, the framework enables fine-grained quantification of environmental impacts. The analysis reveals that increasing thread count generally reduces total environmental footprints, albeit with diminishing marginal returns; while carbon footprints are predominantly driven by operational phases, water footprints are largely dominated by embodied impacts. By jointly incorporating both footprint types at the job granularity, this work establishes a novel paradigm for assessing HPC sustainability.

This study investigates whether internal circuits in language models exhibit task-specificity and consistency, and how such properties inform our understanding of—and ability to intervene on—model behavior. Employing edge attribution patching and component ablation, the authors systematically evaluate causally critical subgraphs within attention heads and MLP layers across six tasks and seven models. Their analysis reveals, for the first time, that circuits within a single task are highly reused and essential for performance, yet circuits across different tasks substantially overlap, with task-exclusive components contributing minimally. This finding challenges the prevailing assumption of task-dedicated circuits and offers a new perspective on model interpretability and targeted intervention.