A Family of Divergence Measures for Evaluating the Reconstruction Quality of Explainable Ensemble Trees

2026-05-19

AI Summary

Existing correlation-based methods struggle to accurately assess the fidelity of interpretable surrogate models in reconstructing the co-occurrence structure of ensemble tree models. This work proposes a consistency–association distinction framework centered on normalized Loss of Interpretability (nLoI), leveraging the Cressie–Read power divergence family (λ=2) to construct a multidimensional evaluation metric. For the first time, nLoI is decomposed into within-node and between-node components, enabling fine-grained diagnosis of error sources. Four complementary metrics are unified under a single permutation test, offering both theoretical boundedness and symmetry. Empirical results demonstrate that the proposed approach rigorously controls Type I error across three benchmark datasets and significantly outperforms existing correlation-based methods in detecting reconstruction gradients.

Abstract

Validating interpretable surrogate models for ensemble learners requires measuring agreement between the ensemble's internal representation and its surrogate approximation, rather than mere association. Correlation-based approaches are scale-invariant and fail to detect systematic discrepancies in co-occurrence structure. We propose a statistical framework grounded in the agreement-association distinction, centered on the normalized Loss of Interpretability (nLoI). Rooted in the Cressie-Read power divergence family with lambda equal to 2, the nLoI admits a closed-form decomposition into within-node and between-node components, providing a unique diagnostic capability to identify precisely where and why reconstruction fails. The framework incorporates four complementary measures capturing distinct structural facets of approximation quality. A unified permutation testing procedure delivers valid inference for all measures within a single resampling pass. Theoretical properties, including boundedness and symmetry, are established for each metric. Monte Carlo simulations and empirical evaluations confirm exact Type I error control and demonstrate that these measures detect reconstruction fidelity gradients invisible to correlation-based alternatives. The framework is developed and illustrated in the context of Explainable Ensemble Trees (E2Tree), and empirical evaluation on three benchmark datasets illustrates the practical utility of the framework.

Problem

• reconstruction quality
• explainable ensemble trees
• interpretability
• divergence measures
• surrogate models

Innovation

• normalized Loss of Interpretability
• Cressie-Read power divergence
• agreement vs. association
• Explainable Ensemble Trees
• permutation testing