大模型推理能力

A Reasoning-Focused Legal Retrieval Benchmark

Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, Daniel E. Ho

机构 * Stanford University（斯坦福大学） ； Princeton University（普林斯顿大学） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments CS&Law 2025. For data, see https://reglab.github.io/legal-rag-benchmarks/

$\texttt{Complex-Edit}$: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmark

Siwei Yang, Mude Hui, Bingchen Zhao, Yuyin Zhou, Nataniel Ruiz, Cihang Xie

专题命中 推理评测 ：CoT(title)；分类 cs.AI

Comments Project Page: https://ucsc-vlaa.github.io/Complex-Edit/, Dataset: https://huggingface.co/datasets/UCSC-VLAA/Complex-Edit

EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos

Yuxuan Li, Vijay Veerabadran, Michael L. Iuzzolino, Brett D. Roads, Asli Celikyilmaz, Karl Ridgeway

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation

Jeff Da, Maxwell Forbes, Rowan Zellers, Anthony Zheng, Jena D. Hwang, Antoine Bosselut, Yejin Choi

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments ACL 2021

SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks

Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, Xian Li

专题命中 推理评测 ：reasoning(title)；分类 cs.LG

Comments 29 pages, 16 figures

MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks

Mirelle Bueno, Roberto Lotufo, Rodrigo Nogueira

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments GenBench Workshop by EMNLP 2024: Camera-ready version

ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation

Ana Brassard, Benjamin Heinzerling, Keito Kudo, Keisuke Sakaguchi, Kentaro Inui

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments 18 pages, 7 figures, accepted to COLM 2024. Data available here: https://github.com/a-brassard/ACORN

Common Sense Reasoning for Deepfake Detection

Yue Zhang, Ben Colman, Xiao Guo, Ali Shahriyari, Gaurav Bharaj

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

eagerlearners at SemEval2024 Task 5: The Legal Argument Reasoning Task in Civil Procedure

Hoorieh Sabzevari, Mohammadmostafa Rostamkhani, Sauleh Eetemadi

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Open-Ended Multi-Modal Relational Reasoning for Video Question Answering

Haozheng Luo, Ruiyang Qin, Chenwei Xu, Guo Ye, Zening Luo

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

Comments 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)

Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository

Ajinkya Deshpande, Anmol Agarwal, Shashank Shet, Arun Iyer, Aditya Kanade, Ramakrishna Bairi, Suresh Parthasarathy

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

Comments Preprint with additional experiments

DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge

Bufang Yang, Siyang Jiang, Lilin Xu, Kaiwei Liu, Hai Li, Guoliang Xing, Hongkai Chen, Xiaofan Jiang, Zhenyu Yan

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoning

Zhouhang Xie, Bodhisattwa Prasad Majumder, Mengjie Zhao, Yoshinori Maeda, Keiichi Yamada, Hiromi Wakaki, Julian McAuley

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Offline Imitation Learning with Variational Counterfactual Reasoning

Bowei He, Zexu Sun, Jinxin Liu, Shuai Zhang, Xu Chen, Chen Ma

专题命中 推理评测 ：reasoning(title)；分类 cs.LG

Comments Published on NeurIPS2023

Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models

Zheyu Zhang, Han Yang, Bolei Ma, David Rügamer, Ercong Nie

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments CoNLL 2023 BabyLM Challenge

Probing Physical Reasoning with Counter-Commonsense Context

Kazushi Kondo, Saku Sugawara, Akiko Aizawa

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments Accepted to ACL 2023(Short Paper)

Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models

Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, Vered Shwartz

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

SODAPOP: Open-Ended Discovery of Social Biases in Social Commonsense Reasoning Models

Haozhe An, Zongxia Li, Jieyu Zhao, Rachel Rudinger

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments EACL 2023

The Legal Argument Reasoning Task in Civil Procedure

Leonard Bongard, Lena Held, Ivan Habernal

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments Camera ready, to appear at the Natural Legal Language Processing Workshop 2022 co-located with EMNLP

Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning

Roshanak Mirzaee, Parisa Kordjamshidi

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)

ConceptNet infused DialoGPT for Underlying Commonsense Understanding and Reasoning in Dialogue Response Generation

Ye Liu, Wolfgang Maier, Wolfgang Minker, Stefan Ultes

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments this is a long paper, the short version was accepted by SemDial 2022

KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning

Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

Comments Accepted to EMNLP 2021 Findings

Diagnostic reasoning with A-Prolog

Marcello Balduccini, Michael Gelfond

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

Comments 46 pages, 1 Postscript figure

Journal ref TPLP Vol 3(4&5) (2003) 425-461

Multimodal Language Models Benchmarked Against the NRC Reactor Operator Licensing Examination: Fine-Tuning and Retrieval Strategies

基于美国核管理委员会反应堆操作员执照考试的多模态语言模型微调与检索策略基准测试

Isak Hwang, Yoon Pyo Lee, Syed Bahauddin Alam

机构 * organization= Department of Nuclear Engineering, Hanyang University , addressline= 222 Wangsimni-ro , postcode= 04763 , state= Seongdong-gu , city= Seoul , country= South Korea ； organization= The Grainger College of Engineering, Nuclear, Plasma \& Radiological Engineering, University of Illinois Urbana-Champaign , city= Urbana , state= IL , country= USA 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract_cn)；分类 cs.CL、cs.AI

AI总结 该研究针对美国核管理委员会反应堆操作员执照考试，评估310亿参数多模态模型应用核知识的能力，通过对比基础模型与多种微调及检索配置，发现固定大小分块RAG的SFT配置表现最佳，并揭示了分块策略规律及RAFT与SFT的性能差异。

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

ReLoop：结构建模与行为验证用于可靠的基于LLM的优化

Junbo Jacob Lian, Yujun Sun, Huiling Chen, Chaoyu Zhang, Hanzhang Qin, Chung-Piaw Teo

机构 * McCormick School of Engineering, Northwestern University（西北大学工程学院） ； Wenzhou Buyi Pharmacy Chain Co., Ltd.（温州-buyi药链有限公司） ； College of Computer Science and Artificial Intelligence, Wenzhou University（温州大学计算机科学与人工智能学院） ； Department of Decision Analytics and Operations, City University of Hong Kong（香港城市大学决策分析与运营部门） ； Institute of Operations Research and Analytics, National University of Singapore（新加坡国立大学运筹学与分析研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

AI总结 ReLoop通过结构生成和行为验证机制，解决LLM生成优化代码的可行性与正确性差距问题，提升代码执行准确性和鲁棒性。

Comments Code and benchmark: https://github.com/junbolian/ReLoop

Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thought

LLMs能否像自动定理证明器一样进行Rust验证？VCoT-Bench：通过验证思维链进行评估

Zichen Xie, Wenxi Wang

机构 * University of Virginia（弗吉尼亚大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

AI总结 本文提出VCoT-Bench，通过验证思维链评估LLMs在Rust验证中的能力，揭示其在不同证明类型和缺失证明情况下的脆弱性。

Comments Accepted at ICML 2026

Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

Avalon-ToM-Bench：通过非对称游戏机制评估细粒度心理理论

Yen-Shan Chen, Yu Chian Duan, Chih-En Kuo, Jian-Bin Wu, Yun-Nung Chen

机构 * National Taiwan University（台湾大学） ； CyCraft AI Lab（CyCraft人工智能实验室） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

AI总结 本文提出Avalon-ToM-Bench基准，基于阿瓦隆游戏机制评估大语言模型的细粒度心理理论，发现模型ToM能力不足源于推理策略而非知识或表征，推理训练比测试时思维链增益更显著。

ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons

ComboShoppingBench：评估大型语言模型智能体在带优惠券的预算约束组合购物篮任务中的表现

Adrian Li, Kelong Mao, Yudong Guo, Heming Xia, Xinwei Yang, Lirui Luo, Jace Wong, Pu Yao, Sulong Xu, Simiu Gu

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

AI总结 本研究推出ComboShoppingBench组合购物基准，通过探索智能体、LLM评判者及确定性验证，发现各类LLM智能体在该基准上表现不佳，凸显其在约束感知组合购物上仍有巨大改进空间。

WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management

WuYuEval：面向固体废物管理的大语言模型多级基准测试

Yi Zhang, Hongyang Wang, Zheng Hao Leong, Zihao Wu, Kaijun Lin, Zhixing Pan, Qixun Huangfu, Wei Ren, Wenyan Wu, Fangyun Wang, Wenting Yu, Hengyu Lin, Muling Yang, Zongguo Wen

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

AI总结 WuYuEval是面向固体废物管理的多级基准测试，含基础与专家模块，评估33个LLM的SWM能力，发现其在专业任务表现差，为开发专用基础模型提供依据。

A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges

大型语言模型有多独立？一种用于审计行为纠缠和重加权验证者集合的统计框架

Chenchen Kuai, Jiwan Jiang, Zihao Zhu, Hao Wang, Keshu Wu, Zihao Li, Yunlong Zhang, Chenxi Liu, Zhengzhong Tu, Zhiwen Fan, Yang Zhou

机构 * Texas A&M University（德克萨斯农工大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；verifier(abstract)；分类 cs.CL、cs.AI

AI总结 本文提出一种统计框架，用于审计大型语言模型之间的行为纠缠，通过多分辨率层次结构和信息理论度量，分析行为纠缠对验证性能的影响，并通过重加权验证者集合减少相关偏差。