gcor: A Python Implementation of Categorical Gini Correlation and Its Inference

2025-06-23

AI Summary

This paper addresses the challenge of measuring dependence between numerical and categorical variables by proposing the Categorical Gini Correlation (CGC)—a novel, theoretically consistent, and computationally efficient dependency measure. CGC is grounded in Gini impurity, offering a clear statistical interpretation, asymptotic normality, and sensitivity to nonlinear and non-monotonic relationships. We derive procedures for constructing confidence intervals and conducting asymptotic independence tests. Furthermore, we design a vectorized, multi-process parallel algorithm that significantly accelerates computation on large-scale datasets. Empirical evaluations demonstrate that CGC outperforms state-of-the-art methods—including MIC, Distance Correlation, and Kendall’s τ-b—in both robustness and discriminative power for feature selection. An open-source Python package is released, enabling end-to-end statistical inference and high-dimensional feature selection.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Graphical Models

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Categorical Gini Correlation (CGC), introduced by Dang et al. (2020), is a novel dependence measure designed to quantify the association between a numerical variable and a categorical variable. It has appealing properties compared to existing dependence measures, such as zero correlation mutually implying independence between the variables. It has also shown superior performance over existing methods when applied to feature screening for classification. This article presents a Python implementation for computing CGC, constructing confidence intervals, and performing independence tests based on it. Efficient algorithms have been implemented for all procedures, and they have been optimized using vectorization and parallelization to enhance computational efficiency.

Problem

• Measure association between numerical and categorical variables
• Implement Python tools for CGC computation and inference
• Optimize algorithms for efficiency via vectorization and parallelization

Innovation

• Python implementation of Categorical Gini Correlation
• Efficient algorithms with vectorization optimization
• Parallelization for enhanced computational efficiency