🤖 AI Summary
This work addresses the challenge of Shannon entropy estimation in realistic settings where the total number of species is unknown, a scenario in which traditional methods fail due to their reliance on a known species count. The authors propose the first application of the Pitman–Yor process to entropy estimation, formulating a Bayesian nonparametric model that treats the underlying distribution as an infinite-dimensional random measure. This approach enables stable and reliable diversity estimation even when observed species constitute only a small fraction of the true total. Theoretical analysis establishes the consistency of the proposed estimator under regularly varying distributions, while numerical experiments demonstrate its robustness and stability across diverse simulation settings. By explicitly accounting for the uncertainty inherent in species richness, the method significantly extends the applicability of entropy estimation to open-population scenarios.
📝 Abstract
The Shannon entropy is a fundamental measure for quantifying diversity and model complexity in fields such as information theory, ecology, and genetics. However, many existing studies assume that the number of species is known, an assumption that is often unrealistic in practice. In recent years, efforts have been made to relax this restriction. Motivated by these developments, this study proposes an entropy estimation method based on the Pitman--Yor process, a representative approach in Bayesian nonparametrics. By approximating the true distribution as an infinite-dimensional process, the proposed method enables stable estimation even when the number of observed species is smaller than the true number of species. This approach provides a principled way to deal with the uncertainty in species diversity and enhances the reliability and robustness of entropy-based diversity assessment. In addition, we investigate the convergence property of the Shannon entropy for regularly varying distributions and use this result to establish the consistency of the proposed estimator. Finally, we demonstrate the effectiveness of the proposed method through numerical experiments.