How Sparsity Allocation Shapes Label-Free Post-Pruning Recoverability
稀疏性分配如何塑造无标签后剪枝恢复能力
Qishi Zhan, Minxuan Hu, Liang He
AI总结 本文研究了在固定激活统计修复后端下,稀疏性分配如何影响后修复恢复能力,通过比较ERK和LAMP分配在不同数据集和模型上的表现,发现分配选择对后修复准确性有显著影响,并揭示了修复敏感的过渡区域。
详情
在高稀疏度下进行无结构幅度剪枝可能导致神经网络精度降至接近随机水平,而在实际部署中可能无法进行带标签的重新训练。无标签后剪枝修复方法可以部分恢复塌陷的稀疏模型,但其有效性取决于上游剪枝分配留下的稀疏模型。本文研究了在固定激活统计修复后端下,稀疏性分配如何影响后修复恢复能力。我们在CIFAR-10、CIFAR-100和ImageNetet上,使用ResNet-18、ResNet-34和ResNet-50,在90%到95.5%的稀疏度下,比较ERK和LAMP分配在相同无标签修复协议下的表现。结果表明,在相同全局稀疏度下,分配选择可以显著改变后修复准确性,并且优选的分配会随着架构、数据集难度和稀疏度水平而变化。我们识别出一个修复敏感的过渡区域,在此区域内批归一化重新校准开始失效,而激活统计修复仍能恢复非平凡的准确性。在ImageNet-100和DenseNet-121上的额外验证表明,此可恢复区域的位置和宽度取决于数据规模和连接结构。这些发现表明,剪枝分配和后剪枝修复应联合研究,因为分配决定了可用于无标签恢复的激活信号量。
Unstructured magnitude pruning at high sparsity can reduce neural network accuracy to near-random performance, while labeled retraining may be unavailable in practical deployment settings. Label-free post-pruning repair methods can partially recover collapsed sparse models, but their effectiveness depends on the sparse model left by the upstream pruning allocation. This paper studies how sparsity allocation shapes post-repair recoverability under a fixed activation-statistic repair backend. We compare ERK and LAMP allocations under the same label-free repair protocol across CIFAR-10, CIFAR-100, and Imagenette with ResNet-18, ResNet-34, and ResNet-50 at sparsities from 90% to 95.5%. The results show that allocation choice can substantially change post-repair accuracy at the same global sparsity, and that the preferred allocation varies with architecture, dataset difficulty, and sparsity level. We identify a repair-sensitive transition regime in which BatchNorm recalibration begins to fail, while activation-statistic repair still recovers nontrivial accuracy. Additional validation on ImageNet-100 and DenseNet-121 shows that the location and width of this recoverable regime depend on data scale and connectivity structure. These findings suggest that pruning allocation and post-pruning repair should be studied jointly, since the allocation determines how much activation signal remains available for label-free recovery.