Asymptotic and finite-sample distributions of one- and two-sample empirical relative entropy, with application to change-point detection

2025-12-18

AI Summary

This paper addresses the sensitivity to distribution shift types and limited statistical power of conventional moment- or information-criterion-based offline change-point detection methods. We propose a novel testing framework grounded in empirical relative entropy. Theoretically, we establish the first finite-sample Berry–Esseen-type error bounds for empirical relative entropy under both one- and two-sample settings; introduce a convexity-driven inequality derivation paradigm tailored to nonlinear summation statistics; and rigorously characterize its asymptotic normality and statistical power. Methodologically, we design an offline change-point algorithm that analytically controls Type-I error while maintaining universal detectability across diverse distribution shifts. Empirically, our method significantly outperforms classical approaches on synthetic data and achieves high-accuracy, robust change-point localization on real-world temperature time series and stock index volatility data.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationData Mining & Knowledge Management: Anomaly/Outlier DetectionReasoning under Uncertainty: Stochastic Optimization

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsSecurity and Privacy: Large-scale security measurementsEconomics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Relative entropy, as a divergence metric between two distributions, can be used for offline change-point detection and extends classical methods that mainly rely on moment-based discrepancies. To build a statistical test suitable for this context, we study the distribution of empirical relative entropy and derive several types of approximations: concentration inequalities for finite samples, asymptotic distributions, and Berry-Esseen bounds in a pre-asymptotic regime. For the latter, we introduce a new approach to obtain Berry-Esseen inequalities for nonlinear functions of sum statistics under some convexity assumptions. Our theoretical contributions cover both one- and two-sample empirical relative entropies. We then detail a change-point detection procedure built on relative entropy and compare it, through extensive simulations, with classical methods based on moments or on information criteria. Finally, we illustrate its practical relevance on two real datasets involving temperature series and volatility of stock indices.

Problem

• Derives asymptotic and finite-sample distributions of empirical relative entropy
• Develops statistical tests for offline change-point detection using relative entropy
• Compares relative entropy-based detection with classical moment-based methods

Innovation

• Empirical relative entropy for change-point detection
• Berry-Esseen bounds under convexity assumptions
• Comparison with moment-based and information criteria methods