What Makes Chain-of-Thought Work at Probe Time? Local Co-occurrence Rather Than Global Derivation
链式思维在探测时为何有效?局部共现而非全局推导
Xiang Wang, Wei Wei
AI总结 研究链式思维提示在探测时提升语言模型准确率的原因,发现增益主要来自词汇激活和短距离标记共现,而非句子级逻辑推导。
详情
链式思维提示可靠地提高了语言模型的准确性,但推理文本的哪些属性驱动了这种改进尚不清楚。先前的工作主要研究生成本身的行为。我们转而提出一个探测时问题:给定上下文中的固定推理文本,该文本中的什么改变了答案?我们确定了增益的两个互补来源。首先,即使是全局词序打乱的推理文本也显著优于无推理基线,表明存在强烈的词汇激活效应。更重要的是,结构化文本带来的额外增益似乎较少来自句子级的逻辑排序,而更多来自短距离标记邻接。保留仅$n^\star{=}2$--$3$个标记的连续窗口即可恢复向完整链式思维性能的大部分剩余增益。支持性实验排除了显式答案声明或答案值的复制以及完整的语法实现作为主要驱动因素。进一步的泛化实验表明,这种定性模式在多个模型家族、参数规模和数据集上保持稳定。这些结果支持探测时链式思维的局部共现激活解释,其中观察到的增益主要来自词汇激活和短距离标记共现,而非句子级逻辑推导。
Chain-of-thought (CoT) prompting reliably improves language-model accuracy, but which properties of a rationale text drive the improvement is poorly understood. Prior work has largely studied generation-time behavior. We instead ask a probe-time question: given a fixed rationale in context, what in that text changes the answer? We identify two complementary sources of the gain. First, even a globally word-shuffled rationale substantially outperforms the no-rationale baseline, indicating a strong lexical activation effect. More importantly, the additional gain from structured text appears to arise less from sentence-level logical ordering and more from short-range token adjacency. Preserving contiguous windows of just $n^\star{=}2$--$3$ tokens recovers most of the remaining gain toward full CoT performance. Supporting experiments rule out copying of explicit answer declarations or answer values, as well as full grammatical realization, as primary drivers. Further generalization experiments show that the qualitative pattern remains stable across multiple model families, parameter scales, and datasets. These results support a local co-occurrence activation (LCA) account of probe-time CoT, in which the observed gains appear to arise primarily from lexical activation and short-range token co-occurrence rather than sentence-level logical derivation.