Ensemble Estimation of Generalized Mutual Information with Applications to Genomics · arXivDesk
1701.08083Jan 27, 2017Published in IEEE Transactions on Information Theory in 2021; 42 pages, 3 figures; a shorter version of this paper was published at IEEE ISIT 2017 under the title "Ensemble estimation of mutual information"
Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
Mutual information is a measure of the dependence between random variables that has been used successfully in myriad applications in many fields. Generalized mutual information measures that go beyond classical Shannon mutual information have also received much interest in these applications. We derive the mean squared error convergence rates of kernel density-based plug-in estimators of general mutual information measures between two multidimensional random variables X and Y for two cases: 1) X and Y are continuous; 2) X and
Nearby in the stack
Y
may have any mixture of discrete and continuous components. Using the derived rates, we propose an ensemble estimator of these information measures called GENIE by taking a weighted sum of the plug-in estimators with varied bandwidths. The resulting ensemble estimators achieve the
1/N
parametric mean squared error convergence rate when the conditional densities of the continuous variables are sufficiently smooth. To the best of our knowledge, this is the first nonparametric mutual information estimator known to achieve the parametric convergence rate for the mixture case, which frequently arises in applications (e.g. variable selection in classification). The estimator is simple to implement and it uses the solution to an offline convex optimization problem and simple plug-in estimators. A central limit theorem is also derived for the ensemble estimators and minimax rates are derived for the continuous case. We demonstrate the ensemble estimator for the mixed case on simulated data and apply the proposed estimator to analyze gene relationships in single cell data.