大模型推理能力

Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?

Sohee Yang, Nora Kassner, Elena Gribovskaya, Sebastian Riedel, Mor Geva

机构 * Google DeepMind（谷歌DeepMind） ； UCL（伦敦大学学院） ； Google Research（谷歌研究） ； Tel Aviv University（特拉维夫大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Findings of ACL 2025

SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

Zixiang Xu, Yanbo Wang, Yue Huang, Jiayi Ye, Haomin Zhuang, Zirui Song, Lang Gao, Chenxi Wang, Zhaorun Chen, Yujun Zhou, Sixian Li, Wang Pan, Yue Zhao, Jieyu Zhao, Xiangliang Zhang, Xiuying Chen

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Code available at https://github.com/xzx34/SocialMaze

SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis

Yansong Li, Paula Branco, Alexander M. Hoole, Manish Marwah, Hari Manassery Koduvely, Guy-Vincent Jourdan, Stephan Jou

机构 * University of Ottawa（渥太华大学） ； OpenText（OpenText公司） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；logical reasoning(abstract)；分类 cs.CL

Journal ref 2025 IEEE Symposium on Security and Privacy (SP), 2025, pp. 2791-2809

ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

Yifan Xiang, Zhenxi Zhang, Bin Li, Yixuan Weng, Shoujun Zhou, Yangfan He, Keqin Li

机构 * Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所） ； School of Engineering, Westlake University（西湖大学工程学院） ； University of Minnesota – Twin Cities（明尼苏达大学双城分校） ； University of Toronto（多伦多大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；CoT(abstract)；分类 cs.AI

Comments Work in progress

CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generation

Ningxin Gui, Qianghuai Jia, Feijun Jiang, Yuling Jiao, dechun wang, Jerry Zhijian Yang

机构 * School of Mathematics and Statistics（数学与统计学学院） ； Alibaba International Digital Commerce（阿里巴巴国际数字商务） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；CoT(abstract)；分类 cs.AI

Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning

Shaun Baek, Shaun Esua-Mensah, Cyrus Tsui, Sejan Vigneswaralingam, Abdullah Alali, Michael Lu, Vasu Sharma, Sean O'Brien, Kevin Zhu

机构 * Emory University（埃默里大学） ； Algoverse AI Research（Algoverse AI研究） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；logical reasoning(abstract)；分类 cs.CL

VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, Xiang Yue

专题命中 推理评测 ：reasoning(title,abstract)；logical reasoning(abstract)；分类 cs.CL

Comments 56 pages, 43 figures

Reasoning with LLMs for Zero-Shot Vulnerability Detection

Arastoo Zibaeirad, Marco Vieira

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.AI

Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoning

Md Rizwan Parvez

专题命中 推理评测 ：reasoning(title,abstract)；CoT(abstract)；分类 cs.CL

Comments Accepted at NAACL KnowledgeNLP 2025

Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents

Rui Xu, MingYu Wang, XinTao Wang, Dakuan Lu, Xiaoyu Tan, Wei Chu, Yinghui Xu

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.AI

Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases

Pengcheng Qiu, Chaoyi Wu, Shuyu Liu, Weike Zhao, Zhuoxia Chen, Hongfei Gu, Chuanjin Peng, Ya Zhang, Yanfeng Wang, Weidi Xie

专题命中 推理评测 ：reasoning(title,abstract)；planning(abstract)；分类 cs.CL

MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps

Xiongtao Zhou, Jie He, Lanyu Chen, Jingyu Li, Haojing Chen, Víctor Gutiérrez-Basulto, Jeff Z. Pan, Hanjie Chen

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments NAACL 2025

Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning

Mohamed Aghzal, Erion Plaku, Ziyu Yao

专题命中 推理评测 ：reasoning(title,abstract)；planning(abstract)；分类 cs.CL

PEA: Enhancing LLM Performance on Computational-Reasoning Tasks

Zi Wang, Shiwei Weng, Mohannad Alhanahnah, Somesh Jha, Tom Reps

专题命中 推理评测 ：reasoning(title,abstract)；planning(abstract)；分类 cs.AI

MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs

Manar Abdelatty, Jingxiao Ma, Sherief Reda

专题命中 推理评测 ：reasoning(title,abstract)；CoT(abstract)；分类 cs.CL

Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild

Wanpeng Hu, Haodi Liu, Lin Chen, Feng Zhou, Changming Xiao, Qi Yang, Changshui Zhang

专题命中 推理评测 ：reasoning(title,abstract)；CoT(abstract)；分类 cs.AI

Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning

Hang Yin, Zhifeng Lin, Xin Liu, Bin Sun, Kan Li

专题命中 推理评测 ：reasoning(title,abstract)；CoT(abstract)；分类 cs.AI

Reasoning-Aware Query-Focused Summarization over Multi-Table Data

Xiaochuan Lin, Xiangyong Chen

专题命中 推理评测 ：reasoning(title,abstract)；logical reasoning(abstract)；分类 cs.CL

Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge

Haotong Zhang

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations

Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, Conghui He

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Equal contribution: Jiaxing Sun, Weiquan Huang, Jiang Wu; Corresponding author: Conghui He

The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights

Wenhao Zhu, Shujian Huang, Fei Yuan, Cheng Chen, Jiajun Chen, Alexandra Birch

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs

Houman Mehrafarin, Arash Eshghi, Ioannis Konstas

专题命中 推理评测 ：reasoning(title,abstract)；logical reasoning(abstract)；分类 cs.CL

Comments To appear in EMNLP Main 2024

MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education

Naiming Liu, Shashank Sonkar, Myco Le, Richard Baraniuk

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted at EMNLP 2024 Findings Track

Towards a Benchmark for Causal Business Process Reasoning with LLMs

Fabiana Fournier, Lior Limonad, Inna Skarbovsky

专题命中 推理评测 ：reasoning(title,abstract)；planning(abstract)；分类 cs.AI

Comments 12 pages, 1 figure

Journal ref NLP4BPM workshop at BPM 2024

Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

Kohei Uehara, Nabarun Goswami, Hanqin Wang, Toshiaki Baba, Kohtaro Tanaka, Tomohiro Hashimoto, Kai Wang, Rei Ito, Takagi Naoya, Ryo Umagami, Yingyi Wen, Tanachai Anakewat, Tatsuya Harada

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies

Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, Ben Athiwaratkun

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Chain-of-Though (CoT) prompting strategies for medical error detection and correction

Zhaolong Wu, Abul Hasan, Jinge Wu, Yunsoo Kim, Jason P. Y. Cheung, Teng Zhang, Honghan Wu

专题命中 推理评测 ：CoT(title,abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments accepted as NAACL workshop

PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset

Arda Uzunoglu, Abdalfatah Rashid Safa, Gözde Gül Şahin

专题命中 推理评测 ：planning(title,abstract)；reasoning(abstract)；分类 cs.CL

Comments 9 pages, ACL 2024 Findings

Keypoint-based Progressive Chain-of-Thought Distillation for LLMs

Kaituo Feng, Changsheng Li, Xiaolu Zhang, Jun Zhou, Ye Yuan, Guoren Wang

专题命中 推理评测 ：chain-of-thought(title,abstract)；reasoning(abstract)；分类 cs.CL

Comments Accepted by ICML 2024

BEnQA: A Question Answering and Reasoning Benchmark for Bengali and English

Sheikh Shafayat, H M Quamran Hasan, Minhajur Rahman Chowdhury Mahim, Rifki Afina Putri, James Thorne, Alice Oh

专题命中 推理评测 ：reasoning(title,abstract)；chain-of-thought(abstract)；分类 cs.CL