Total/dual correlation/coherence, redundancy/synergy, complexity, and O-information for real and complex valued multivariate data

2025-07-11

AI Summary

This work addresses theoretical limitations of multivariate information measures under Gaussian assumptions and the need for structured analysis. Methodologically, it derives unified analytical expressions for total correlation (TC), dual total correlation (DTC), O-information, TSE complexity, and the redundancy–synergy index (RSI); introduces structured O-information to explicitly model inter-group synergy; establishes that DTC equals the KL divergence between the standardized inverse covariance matrix and the identity matrix under the inverse Wishart distribution—endowing it with statistical testability; and generalizes all measures to elliptical distributions and complex-valued data. Contributions include: (i) precise disentanglement of intra-group redundancy from inter-group synergy; (ii) quantification of how variable pairwise connections shape global information properties; and (iii) an open-source computational framework, empirically validated on structured population-level analyses. (149 words)

Technology Category

Machine Learning: Information TheoryConstraint Satisfaction and Optimization: Distributed CSP/OptimizationKnowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

Security and Privacy: Large-scale security measurementsWeb Mining and Content Analysis: Bridging structured and unstructured dataGraph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Firstly, assuming Gaussianity, equations for the following information theory measures are presented: total correlation/coherence (TC), dual total correlation/coherence (DTC), O-information, TSE complexity, and the redundancy-synergy index (RSI). Since these measures are functions of the covariance matrix "S" and its inverse "S^-1", the associated Wishart and inverse-Wishart distributions are of note. The DTC is shown here to be the Kullback-Leibler (KL) divergence for the inverse-Wishart pair "(S^-1)" and its diagonal matrix "diag(S^-1)", shedding light on its interpretation as a measure of "total partial correlation", -lndetP, with test hypothesis H0: P=I, where "P" is the standardized inverse covariance (i.e. P=(D^-1/2)(S^-1)(D^-1/2), with D=diag(S^-1)). The second aim of this paper introduces a generalization of all these measures for structured groups of variables. For instance, consider three or more groups, each consisting of three or more variables, with predominant redundancy within each group, but with synergistic interactions between groups. O-information will miss the between group synergy (since redundancy occurs more often in the system). In contrast, the structured O-information measure presented here will correctly report predominant synergy between groups. This is a relevant generalization towards structured multivariate information measures. A third aim is the presentation of a framework for quantifying the contribution of "connections" between variables, to the system's TC, DTC, O-information, and TSE complexity. A fourth aim is to present a generalization of the redundancy-synergy index for quantifying the contribution of a group of variables to the system's redundancy-synergy balance. Finally, it is shown that the expressions derived here directly apply to data from several other elliptical distributions. All program codes, data files, and executables are available.

Problem

• Develops Gaussian-based equations for multivariate information theory measures.
• Generalizes measures for structured variable groups to capture synergy.
• Quantifies variable connections' impact on system complexity metrics.

Innovation

• Gaussian-based equations for information theory measures
• Generalization for structured groups of variables
• Framework for quantifying variable connections' contributions