Estimation of mutual information

Abstract

Geometric partition entropy is a recently developed non-parametric approach for entropy estimation, which gives a different way of looking at the problem by taking into account the effect of informative outliers through geometric structures. However, the initial definition was limited only to one-dimensional state spaces. In this work, we considered a generalised version of geometric partition entropy for samples lying in a bounded region of an n-dimensional vector space. This generalized definition is based on the concept of a Voronoi diagram, where the entropy is computed using the measures of the corresponding geometric partitions. Although this approach is theoretically meaningful, constructing Voronoi diagrams becomes computationally expensive and often unreliable as the dimension increases. To overcome this issue, approximation methods have been developed that speed up computation while maintaining the same accuracy. Among these, the partition intersection (⇡) approximation is particularly useful, as it allows direct estimation of marginal entropy and can also be used to build an efficient mutual information estimator. Further, to handle high-dimensional data more effectively, a granular ball based entropy estimator is proposed. Instead of relying on complicated geometric partitions, this method uses adaptive ball structures to represent the data. This method not only reduces computational cost but also makes the estimator more robust to outliers and scalable to higher dimensions. Overall, this thesis contributes to develop a new estimator for mutual information by adding the granular ball computing theory.