大模型推理能力 - arXivDaily 专题

2606.19350 2026-06-19 cs.CL 新提交 85%

Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

基于因果归因的剪枝保留大型语言模型的推理性能

Amogh Sheth, Biruk Assefa, Yi Wen Huang, Andrew Lin, Yuhao Ge

发表机构 * Edison Academy Magnet School（爱迪生学院磁石学校）； Massachusetts Institute of Technology（麻省理工学院）； State University of New York College at Plattsburgh（纽约州立大学普拉茨堡学院）； The University of Texas at Austin（德克萨斯大学奥斯汀分校）； Independent Researcher（独立研究员）

专题命中其他推理：因果归因剪枝保留推理性能

AI总结提出无需训练的因果归因剪枝（CAP）方法，通过测量注意力头对推理任务的因果影响进行细粒度剪枝，在20%稀疏度下相比Wanda在ARC-Challenge上准确率提升高达61%。

Comments Accepted at the ICLR 2026 Workshop on LLM Reasoning. 13 pages, 2 figures

详情

AI中文摘要

大型语言模型（LLMs）在多步推理方面表现出色，但推理成本高昂。我们引入了因果归因剪枝（CAP），一种无需训练的方法，通过测量注意力头对推理任务的因果影响来识别关键注意力头，并利用这些头级分数指导细粒度的权重剪枝。对于每个注意力头，CAP估计在推理问题的小型校准集上前向传播时掩码该头所导致的预期性能下降。这些因果分数随后被转换为对应投影矩阵的权重级重要性值。与仅基于幅度或激活的标准不同，CAP的干预测量直接捕捉每个头的功能贡献，在20%稀疏度下，相比Wanda在ARC-Challenge上获得高达61%的相对准确率提升。我们在GSM8K、StrategyQA和ARC-Challenge上使用Llama-3-8B-Instruct和Mistral-7B-Instruct在10%、20%和50%稀疏度下评估CAP。在中等稀疏度（10-20%）下，CAP在大多数模型-基准配置中优于Wanda，尤其在Llama-3的ARC-Challenge上提升显著。我们的结果表明，在相同稀疏度下，注意力头级因果归因比相关性剪枝标准能更好地保留下游基准的推理性能，但在50%稀疏度下仍受限于粗粒度的MLP归因。

英文摘要

Large language models (LLMs) excel at multi-step reasoning but incur substantial inference cost. We introduce Causal Attribution Pruning (CAP), a training-free method that identifies critical attention heads by measuring their causal impact on reasoning tasks and uses these head-level scores to guide fine-grained weight pruning. For each attention head, CAP estimates the expected performance degradation when the head is masked during forward passes on a small calibration set of reasoning problems. These causal scores are then converted into weight-level importance values for the corresponding projection matrices. Unlike magnitude-only or activation-based criteria, CAP's interventional measurement directly captures each head's functional contribution, yielding relative accuracy gains of up to 61% over Wanda on ARC-Challenge at 20% sparsity. We evaluate CAP on GSM8K, StrategyQA, and ARC-Challenge using Llama-3-8B-Instruct and Mistral-7B-Instruct at 10%, 20%, and 50% sparsity. At moderate sparsity (10-20%), CAP improves over Wanda in most model-benchmark configurations. with especially large gains on ARC-Challenge for Llama-3. Our results suggest that attention-head-level causal attribution can better preserve reasoning performance on downstream benchmarks than correlational pruning criteria at equivalent sparsity, while remaining limited by coarse MLP attribution at 50% sparsity.

URL PDF HTML ☆

赞 0 踩 0