redundancy measurement

This paper addresses the challenge of accurately quantifying complex (e.g., nonlinear, multimodal, non-functional) dependencies between continuous variables. We propose Predictive Dependence (PD), a fully nonparametric and interpretable dependence measure defined as the expected relative loss in predictive accuracy for $Y$ when ignoring $X$, bounded in $[0,1]$ and identically zero under independence. PD is grounded in kernel density estimation and conditional distribution reconstruction, augmented by Monte Carlo approximation and adaptive bandwidth selection—ensuring rigorous statistical guarantees and exact independence detection. Extensive evaluation across over 90,000 real-world and synthetic datasets demonstrates that PD consistently outperforms state-of-the-art methods—including HSIC and distance correlation—with particularly pronounced gains in weakly nonlinear and multimodal dependency settings.

Existing probabilistic forecasting evaluation metrics primarily emphasize predictive accuracy while neglecting their practical utility in downstream decision-making tasks, leading to a misalignment between evaluation and application. To address this, we propose a data-driven evaluation alignment framework that formulates the learning of a surrogate evaluation function as an end-to-end optimization problem. Leveraging proper scoring rule theory, our approach employs a neural network-parameterized weighted scoring rule to automatically learn an evaluation function aligned with downstream objectives—without assuming any prior cost structure. This work is the first to formalize evaluation alignment as a learnable problem, combining theoretical rigor with engineering scalability. Experiments on synthetic and real-world regression tasks demonstrate its effectiveness: it significantly reduces the gap between evaluation scores and downstream decision utility, enabling rapid, task-adaptive model selection and hyperparameter tuning.

Quantifying and comparing statistical dependence between random variables X and Y across distinct sample sets is challenging under non-stationary distributions, as conventional dependence measures—such as mutual information—lack comparability under distributional shift. Method: We propose Cross Mutual Information (CMI), the first formally defined, cross-sample comparable dependence measure. CMI jointly estimates the joint and marginal distributions from two independent samples to assess consistency in the X–Y dependence structure, thereby overcoming the fundamental incomparability of standard mutual information under distribution shift. Contribution/Results: We establish a tight theoretical connection between CMI and the coefficient of determination (R²) in linear regression. In extensive simulations across diverse nonlinear dependence structures, CMI consistently outperforms baseline methods and exhibits robustness to distribution drift. This framework provides an interpretable, generalizable tool for cross-subject or cross-task dependence modeling in high-dimensional non-stationary domains such as neuroimaging.

Downstream probing only assesses task-relevant information in representations, failing to characterize critical properties—such as equivariance, invariance, and disentanglement—that govern interpretability and generalization; moreover, existing evaluation frameworks lack standardization, modularity, and cross-modal applicability. Method: We propose the first representation quality assessment framework that transcends downstream tasks, employing controlled factorial probe design to systematically quantify informativeness, equivariance, invariance, and disentanglement. The framework is modular, interpretable, and supports cross-modal analysis (e.g., image and speech). Contribution/Results: It establishes the first standardized, multi-dimensional semantic attribute disentanglement protocol. Experiments reveal substantial divergence in intrinsic representation properties—even among models with comparable downstream performance—enabling fine-grained representation understanding, diagnosis, and optimization. This work introduces a novel paradigm and practical toolkit for representation evaluation beyond task-specific metrics.

Existing segmentation evaluation metrics often lack transparency and modularity, making them ill-suited for diverse tasks such as transparent object, specular surface, or lesion segmentation. This work proposes a unified evaluation framework that decomposes metrics into five modular components: prediction representation, target extraction, target matching, score computation, and metric reporting. For the first time, it systematically analyzes the implicit assumptions and design limitations of mainstream binary segmentation metrics through this modular lens. The framework enables task-aware customization of evaluation protocols, reveals evolutionary trajectories among existing metrics, and is accompanied by an open-source toolkit. By offering a principled and interpretable foundation, this approach paves the way for developing more rational and adaptable segmentation evaluation methodologies.

Current performance evaluation metrics—such as accuracy and F1 score—are typically reported as point estimates, ignoring the uncertainty induced by data clustering structures. This oversight often leads to underestimation of variability and potentially misleading model comparisons. To address this, this work proposes a unified framework that expresses a broad class of performance metrics as smooth functionals of the confusion matrix probabilities. By integrating a cluster-robust sandwich variance estimator, the framework enables valid confidence interval construction, hypothesis testing, and paired model comparison. It represents the first systematic application of cluster-robust inference to predictive performance evaluation, accommodating both binary and multiclass settings, and further provides asymptotic theory–based methods for power and sample size calculations. Simulations demonstrate that the proposed approach achieves near-nominal coverage across diverse dependence structures and substantially outperforms conventional methods that ignore clustering; real-data analyses confirm that accounting for clustering can materially alter evaluation conclusions.

This work addresses the misalignment between conventional uncertainty quantification metrics—such as negative log-likelihood and expected calibration error—and the utility of downstream decision-making, which often renders them poor proxies for real-world decision value. To bridge this gap, the paper introduces a “decision-aligned” evaluation principle, systematically exposing the mismatch between widely used scoring rules and common decision tasks. Building on decision theory and proper scoring rules, the authors propose a class of prior-weighted utility-based metrics that directly reflect the impact of predictive uncertainty on decision outcomes. Empirical evaluations across multiple benchmarks and real-world scenarios demonstrate that the proposed metric consistently correlates strongly with actual decision utility, significantly outperforming traditional approaches and offering a principled foundation for decision-relevant uncertainty assessment.

This work addresses the challenge that existing calibration tests for conditional quantile predictors struggle to handle distributional shifts and discrepancies in information sets, lacking feature-aware, continuous monitoring capabilities. The authors propose a distribution-free, game-theoretic sequential auditing framework that formally defines conditional quantile calibration under varying feature information sets—a notion not previously established—and provides finite-time detection guarantees without requiring independent and identically distributed data. By integrating contextual linear betting strategies with nonparametric e-processes, the method enables interpretable, feature-level calibration audits. Empirical evaluations demonstrate that the framework effectively detects significant miscalibration in state-of-the-art time series models, such as Chronos-2, across critical features.

Current evaluations of large language models (LLMs) predominantly focus on point estimation tasks, offering limited insight into their ability to express uncertainty in continuous numerical predictions. This work addresses this gap by introducing prediction intervals into LLM evaluation and proposes QuantSightBench, a novel benchmark that systematically assesses models’ scale awareness, confidence consistency, and calibration across diverse domains using two key metrics: coverage accuracy and interval sharpness. Experimental results reveal that none of the 11 state-of-the-art models achieve the target 90% coverage rate; the best-performing model, Gemini 3.1 Pro, attains only 79.1%. Moreover, calibration significantly degrades under extreme values, exposing a pervasive overconfidence issue among current LLMs.

This study investigates why models with comparable predictive performance exhibit significant differences in the feasibility and proximity of their counterfactual explanations. By fixing a pretrained encoder and varying only the linear classification head, and by integrating standardized local search probes with geometric analysis of the representation space, the work demonstrates that counterfactual behavior constitutes a dimension distinct from predictive accuracy. The findings reveal that the interplay between the decision boundary geometry and local data support jointly determines counterfactual feasibility. Leveraging this insight enables improved counterfactual generation within a fixed model architecture without compromising predictive performance.