Accurate Estimation of Mutual Information in High Dimensional Data
高维数据中互信息的准确估计
Eslam Abdelaleem, K. Michael Martini, Ilya Nemenman
AI总结 针对高维欠采样下互信息估计难题,提出基于低维潜在表示的神经估计器,结合统计一致性检验、偏差校正和置信区间,并引入VSIB概率批评器族,在合成与真实图像数据上实现可靠估计。
Comments 15 pages main text, 21 pages SI, 12 Figs overall
详情
互信息(MI)量化变量之间的统计依赖性,广泛应用于科学领域,但从有限数据中准确估计仍然非常困难。常见方法在现代实验典型的高维欠采样场景($N \lesssim K$)中失败,且没有公认的测试来检测基于神经网络的估计器何时失效,使其实际上无法作为科学仪器使用。我们证明,当统计依赖关系具有低维潜在表示时,神经MI估计器可以变得可靠。样本复杂度由潜在维度$K_Z \ll K$而非环境维度决定——我们通过随机矩阵理论从经验上确认并从理论上奠定了这一机制转变。基于这一见解,我们开发了一个实用协议,为神经估计器提供显式的统计一致性检查、偏差校正和置信区间。此外,我们引入了一类新的概率批评器(VSIB族),在标准估计器失效的高MI值下显著降低偏差和方差。我们在合成基准($K=500$,$N$低至256)、Czyz等人(2023)的标准40数据集基准套件、噪声MNIST($K=784$)以及使用ResNet-20骨干网络的CIFAR-10/100($K=3072$)上验证了该协议。我们的协议始终匹配或超越现有方法,同时是唯一报告置信区间并标记不可靠估计的方法,在真实图像上实现了远低于环境像素维度的可靠MI检测。
Mutual information (MI) quantifies statistical dependence between variables and is widely used across scientific disciplines, yet accurate estimation from finite data remains notoriously difficult. Common approaches fail in high-dimensional, undersampled regimes ($N \lesssim K$) typical of modern experiments, and no accepted tests exist to detect when neural network-based estimators fail, making them effectively unusable as scientific instruments. We show that neural MI estimators can be made reliable when the statistical dependencies admit a low-dimensional latent representation. Sample complexity is then governed by the latent dimensionality $K_Z \ll K$ rather than the ambient dimension -- a regime shift we confirm empirically and ground theoretically via random matrix theory. Building on this insight, we develop a practical protocol that provides neural estimators with explicit statistical consistency checks, bias correction, and confidence intervals. We additionally introduce a new class of probabilistic critics (the VSIB family) that substantially reduce bias and variance at higher MI values where standard estimators break down. We validate the protocol on synthetic benchmarks ($K=500$, $N$ as low as $256$), on the standard 40-dataset benchmark suite of Czyz et al. (2023), on noisy MNIST ($K=784$), and on CIFAR-10/100 ($K=3072$) with a ResNet-20 backbone. Our protocol consistently matches or exceeds existing methods while being the only approach to report confidence intervals and flag unreliable estimates, achieving reliable MI detection well below the ambient pixel dimension on real images.