大模型推理能力

Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning

Yuelyu Ji, Zeshui Yu, Yanshan Wang

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

OpenPI2.0: An Improved Dataset for Entity Tracking in Texts

Li Zhang, Hainiu Xu, Abhinav Kommula, Chris Callison-Burch, Niket Tandon

专题命中 推理评测 ：chain-of-thought(abstract)；planning(abstract)；分类 cs.CL

Comments In EACL 2024

CMMLU: Measuring massive multitask language understanding in Chinese

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, Timothy Baldwin

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, Feng Zhao

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

Comments Project: https://open-compass.github.io/T-Eval

Speech Translation with Large Language Models: An Industrial Practice

Zhichao Huang, Rong Ye, Tom Ko, Qianqian Dong, Shanbo Cheng, Mingxuan Wang, Hang Li

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments Technical report. 13 pages. Demo: https://speechtranslation.github.io/llm-st/

FELM: Benchmarking Factuality Evaluation of Large Language Models

Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, Junxian He

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted by NeurIPS 2023 Track on Datasets and Benchmarks

FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models

Yimin Jing, Renren Jin, Jiahao Hu, Huishi Qiu, Xiaohua Wang, Peng Wang, Deyi Xiong

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments Work in progress

CompCodeVet: A Compiler-guided Validation and Enhancement Approach for Code Dataset

Le Chen, Arijit Bhattacharjee, Nesreen K. Ahmed, Niranjan Hasabnis, Gal Oren, Bin Lei, Ali Jannesari

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.LG

LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation

Shengqiang Zhang, Philipp Wicke, Lütfi Kerem Şenel, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Barbara Plank, Hinrich Schütze

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

Comments 6 pages, 4 figures. The video and code of LoHoRavens are available at https://cisnlp.github.io/lohoravens-webpage/

Answering Unseen Questions With Smaller Language Models Using Rationale Generation and Dense Retrieval

Tim Hartill, Diana Benavides-Prado, Michael Witbrock, Patricia J. Riddle

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Extractive Summarization via ChatGPT for Faithful Summary Generation

Haopeng Zhang, Xiao Liu, Jiawei Zhang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Findings of EMNLP 2023

Guiding Large Language Models via Directional Stimulus Prompting

Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, Xifeng Yan

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted by NeurIPS2023. The code and data are available at https://github.com/Leezekun/Directional-Stimulus-Prompting

A Closer Look into Automatic Evaluation Using Large Language Models

Cheng-Han Chiang, Hung-yi Lee

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments EMNLP 2023 findings (short paper). Code: https://github.com/d223302/A-Closer-Look-To-LLM-Evaluation/

Aligning Large Language Models for Clinical Tasks

Supun Manathunga, Isuru Hettigoda

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments 10 pages, 3 figures

Solving and Generating NPR Sunday Puzzles with Large Language Models

Jingmiao Zhao, Carolyn Jane Anderson

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments To appear in the Proceedings of the 14th International Conference on Computational Creativity (ICCC)

IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

Wenhao Yu, Meng Jiang, Peter Clark, Ashish Sabharwal

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation

Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, Yue Zhang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding

Sayan Ghosh, Shashank Srivastava

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments ACL 2022

Evaluation of LLMs for Process Model Analysis and Optimization

Akhil Kumar, Jianliang Leon Zhao, Om Dobariya

专题命中 推理评测 ：reasoning(abstract,comments)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 5 tables, 4 figures; full research paper currently under review for the Workshop on Information Technologies and Systems (WITS) 2025. The paper presents a comprehensive evaluation of large language models (LLMs) for business process model analysis and optimization, including error detection, reasoning, and scenario-based redesign

Rethinking Human Preference Evaluation of LLM Rationales

Ziang Li, Manasi Ganti, Zixian Ma, Helena Vasconcelos, Qijia He, Ranjay Krishna

机构 * University of Washington（华盛顿大学） ； Stanford University（斯坦福大学） 纠错

专题命中 推理评测 ：reasoning(abstract,comments)；分类 cs.CL、cs.AI；planning(comments)

Comments Published in the XLLM-Reason-Plan Workshop on the Application of LLM Explainability to Reasoning and Planning at COLM 2025

Journal ref Proceedings of the XLLM-Reason-Plan Workshop, Conference on Language Modeling (COLM) 2025

Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations

Yanshu Li

专题命中 推理评测 ：reasoning(abstract,comments)；分类 cs.CL、cs.AI；planning(comments)

Comments Accepted by ICLR 2025 Workshop on Reasoning and Planning for LLMs, 25 pages, 13 tables

Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs

Rohit Saxena, Aryo Pradipta Gema, Pasquale Minervini

专题命中 推理评测 ：reasoning(abstract,comments)；分类 cs.CL、cs.AI；planning(comments)

Comments Accepted at the ICLR 2025 Workshop on Reasoning and Planning for Large Language Models

AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

AI4AI-Bench：用于递归自我改进算法设计中大语言模型智能体的基准测试

Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na

机构 * Navers Lab（Navers实验室） ； Tsinghua University（清华大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本文提出AI4AI-Bench基准测试，测试大语言模型智能体设计训练算法的能力，实验显示现有智能体仅缩小了已有算法与最优值间不到五分之一的差距，发布相关资源供重复测量。

Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis

面向单步逆合成的化学合理性感知大语言模型训练

Bogdan Zagribelnyy, Ivan Ilin, Nikita Bondarev, Maksim Kuznetsov, Mathieu Reymond, Vladimir Aladinskiy, Alex Aliper, Alex Zhavoronkov

机构 * Insilico Medicine AI Limited（英矽智能人工智能有限公司） ； Insilico Medicine Canada Inc.（英矽智能加拿大公司） ； Insilico Medicine Hong Kong Ltd.（英矽智能香港有限公司） 纠错

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 该研究针对单步逆合成的一对多特性，提出Top-K提示范式，构建超大规模反应数据集训练C3LM，结合特定奖励机制在基准上达最优性能，为逆合成系统提供新方向。

EVADE: Evidence-Verified Agentic Diagnosis with Escape

EVADE：带弃权机制的证据验证智能诊断方法

Mohaimenul Azam Khan Raiaan, Nur Mohammad Fahad

机构 * Monash University（莫纳什大学） ； Murdoch University（默多克大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

AI总结 该研究针对医学VLMs不可靠问题，提出无需训练的EVADE方法，通过跨图像视图验证一致性并引入弃权机制，在多医学VQA数据集上提升了校准度与选择性风险，同时维持了准确率。

Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs

超越正确性：混合思维多模态大语言模型（MLLM）的响应行为基准测试与对齐

Xinming Wang, Weinong Wang, Hongming Yang, Yansong Lin, Zheng Ruan, Shangpin Peng, Qiming Peng, Nan Qiao, Fengyuan Lu, Guoqing Ma, Marito Li, Songyang Zhang, Saiyong Yang, Han Hu, Yonglong Tian, Xu-Yao Zhang

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； Large Language Model Department, Tencent（腾讯大语言模型部） ； University of Electronic Science and Technology of China（电子科技大学） ； Hong Kong University of Science and Technology（香港科技大学） ； Zhongguancun Academy（中关村学院） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

AI总结 该研究针对混合思维 MLLM 的思维与非思维模式响应错位问题，构建 PatternEval 基准并开发 PatternRL 方法，可减轻跨模式错位且任务性能损失极小。

Comments 8 tables and 6figures

Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluation

基于基础模型的具身智能体的安全性：攻击面、攻击、防御与评估

Jiawei Liu, Jiacheng Guo, Tian Zhang, Yiwei Xu, Juan Wang, Jinlin Fan, Bowen Xiao

机构 * Wuhan University（武汉大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)

AI总结 该研究以信任边界为核心，针对基于基础模型的具身智能体安全，划分了五个层级与十二个攻击面，分析了58种攻击、61种防御的现状，指出部分领域研究不足并提出开放挑战。

MobileMem: Learning from a Year of Mobile Experiences

MobileMem：从一年的移动体验中学习

Xinle Deng, Yida Xue, Xiangyuan Ru, Yijun Chen, Buqiang Xu, Mingjun Mao, Xinjie Liu, Haoming Xu, Shuofei Qiao, Mengru Wang, Chen Jiang, Yuchen Eleanor Jiang, Lizhong Wang, Jason Wang, Li Zeng, Haofen Wang, Guilin Qi, Huajun Chen, Ningyu Zhang

机构 * OPPO ； OpenKG 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 该研究推出MobileMem基准与框架，基于一年移动体验构建，用于设备端长期记忆研究，支持多类推理，推动智能体从信息检索向体验智能发展。

Comments Technical Report; Project Page: http://mobilemem.openkg.cn/