大模型推理能力

IntentionESC: An Intention-Centered Framework for Enhancing Emotional Support in Dialogue Systems

Xinjie Zhang, Wenxuan Wang, Qin Jin

机构 * School of Information, Renmin University of China（中国人民大学信息学院） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments ACL2025 findings

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICML 2025

Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models

Haoyang Li, Xuejia Chen, Zhanchao XU, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Qing Li, Lei Chen

机构 * Hong Kong Polytechnic University（香港理工大学） ； Hong Kong University of Science and Technology（香港科技大学） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2025

Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models

Fardin Ahsan Sakib, Ziwei Zhu, Karen Trister Grace, Meliha Yetisgen, Ozlem Uzuner

机构 * Department of Computer Science（计算机科学系） ； School of Nursing（护理学院） ； Department of Information Sciences and Technology（信息科学与技术系） ； Department of Biomedical Informatics & Medical Education（生物医学信息学与医学教育系） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias

Ruiquan Huang, Yingbin Liang, Jing Yang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.LG

Comments accepted by ICML 2025

TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding

Max Ku, Thomas Chong, Jonathan Leung, Krish Shah, Alvin Yu, Wenhu Chen

机构 * University of Waterloo（滑铁卢大学） ； Votee AI ； Vector Institute（向量研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments accepted to ACL 2025 main, camera ready

Language Models Benefit from Preparation with Elicited Knowledge

Jiacan Yu, Hannah An, Lenhart K. Schubert

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

LLMs Outperform Experts on Challenging Biology Benchmarks

Lennart Justen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, Shafiq Joty

专题命中 推理评测 ：reasoning(abstract)；math reasoning(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025. The first two authors contributed equally. The codebase is at https://github.com/SalesforceAIResearch/jetts-benchmark

Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis

Kaikai Zhao, Zhaoxiang Liu, Xuejiao Lei, Jiaojiao Zhao, Zhenhong Long, Zipeng Wang, Ning Wang, Meijuan An, Qingliang Meng, Peijun Yang, Minjie Hua, Chaoyang Ma, Wen Liu, Kai Wang, Shiguo Lian

机构 * Unicom Data Intelligence（中国unicom数据智能） ； Data Science & Artificial Intelligence Research Institute（数据科学与人工智能研究院） ； China Unicom（中国unicom） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI、cs.LG

Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

Jadon Geathers, Yann Hicke, Colleen Chan, Niroop Rajashekar, Justin Sewell, Susannah Cornes, Rene F. Kizilcec, Dennis Shung

机构 * Cornell University（康奈尔大学） ； Yale University（耶鲁大学） ； University of California San Francisco（旧金山大学） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 12 pages + 3 pages of references, 4 figures

Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

Yijie Hong, Xiaofei Yin, Xinzhong Wang, Yi Tu, Ya Guo, Sufeng Duan, Weiqiang Wang, Lingyong Fang, Depeng Wang, Huijia Zhu

机构 * Shanghai Jiao Tong University（上海交通大学） ； Ant Security Lab, Ant Group（蚂蚁集团安全实验室） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 13 pages, 3 figures

ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition

Haidar Khan, Hisham A. Alyahya, Yazeed Alnumay, M Saiful Bari, Bülent Yener

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition

Hisham A. Alyahya, Haidar Khan, Yazeed Alnumay, M Saiful Bari, Bülent Yener

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

The Future of MLLM Prompting is Adaptive: A Comprehensive Experimental Evaluation of Prompt Engineering Methods for Robust Multimodal Performance

Anwesha Mohanty, Venkatesh Balavadhani Parthasarathy, Arsalan Shahid

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDL

Heng Ping, Shixuan Li, Peiyu Zhang, Anzhe Cheng, Shukai Duan, Nikos Kanakaris, Xiongye Xiao, Wei Yang, Shahin Nazarian, Andrei Irimia, Paul Bogdan

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures

Yoo Yeon Sung, Hannah Kim, Dan Zhang

专题命中 推理评测 ：reasoning(abstract)；verifier(abstract)；分类 cs.CL、cs.AI

General Scales Unlock AI Evaluation with Explanatory and Predictive Power

Lexin Zhou, Lorenzo Pacchiardi, Fernando Martínez-Plumed, Katherine M. Collins, Yael Moros-Daval, Seraphina Zhang, Qinlin Zhao, Yitian Huang, Luning Sun, Jonathan E. Prunty, Zongqian Li, Pablo Sánchez-García, Kexin Jiang Chen, Pablo A. M. Casares, Jiyun Zu, John Burden, Behzad Mehrbakhsh, David Stillwell, Manuel Cebrian, Jindong Wang, Peter Henderson, Sherry Tongshuang Wu, Patrick C. Kyllonen, Lucy Cheke, Xing Xie, José Hernández-Orallo

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Text2Zinc: A Cross-Domain Dataset for Modeling Optimization and Satisfaction Problems in MiniZinc

Akash Singirikonda, Serdar Kadioglu, Karthik Uppuluri

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Bot Wars Evolved: Orchestrating Competing LLMs in a Counterstrike Against Phone Scams

Nardine Basta, Conor Atkins, Dali Kaafar

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Journal ref Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2025

Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting

Jiyue Jiang, Pengan Chen, Jiuming Wang, Dongchen He, Ziqin Wei, Liang Hong, Licheng Zong, Sheng Wang, Qinze Yu, Zixian Ma, Yanyu Chen, Yimin Fan, Xiangyu Shi, Jiawei Sun, Chuan Wu, Yu Li

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkers

Angel Yahir Loredo Lopez, Tyler McDonald, Ali Emami

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 5 pages (excluding references), Published at Coling 2025, Best Dataset Paper Award

Navigating the Path of Writing: Outline-guided Text Generation with Large Language Models

Yukyung Lee, Soonwon Ka, Bokyung Son, Pilsung Kang, Jaewook Kang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments NAACL 2025 (Industry Track)

Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models

You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 21 pages, 8 figures

LLMs to Support a Domain Specific Knowledge Assistant

Maria-Flavia Lovin

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

PokerBench: Training Large Language Models to become Professional Poker Players

Richard Zhuang, Akshat Gupta, Richard Yang, Aniket Rahane, Zhengyu Li, Gopala Anumanchipalli

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments AAAI 2025

SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

Shraman Pramanick, Rama Chellappa, Subhashini Venugopalan

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2024, Datasets & Benchmarks track

G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o

Tony Cheng Tong, Sirui He, Zhiwen Shao, Dit-Yan Yeung

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queries

Tao Wu, Chuhao Zhou, Yen Heng Wong, Lin Gu, Jianfei Yang

专题命中 推理评测 ：reasoning(abstract)；self-correction(abstract)；分类 cs.AI、cs.LG

HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics

Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, Michael P. Brenner

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

Comments Code and the HARDMath dataset is available at https://github.com/sarahmart/HARDMath