大模型推理能力

Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?

Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang

机构 * York University（约克大学） ； University of Alberta（阿尔伯塔大学） ； Nanyang Technological University（南洋理工大学） ； Salesforce AI Research（Salesforce AI研究） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

Comments Accepted at ACL 2025 Industry Track

M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding

Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li, Shoujun Zhou, Hongliang Li, Fuxia Yang

机构 * Beijing Institute of Technology（北京理工大学） ； The Hong Kong Polytechnic University（香港理工大学） ； Chinese Academy of Sciences（中国科学院） ； Air Force Hospital of Southern Theater Command of PLA（中国人民解放军南部战区空军医院） ； Shenzhen Traditional Chinese Medicine Hospital（深圳中医药医院） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments 19 pages, 8 figures, 7 tables

MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning

Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, Lichao Sun

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments 39 pages, 28 figures, 4 tables

Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories

Jhouben Cuesta-Ramirez, Samuel Beaussant, Mehdi Mounsif

机构 * Akkodis Research（Akkodis研究院） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.LG

Comments Accepted to KONVENS 2025

Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It

Seyed Mahed Mousavi, Edoardo Cecchinato, Lucia Hornikova, Giuseppe Riccardi

机构 * Masaryk University（马萨里克大学） ； Signals and Interactive Systems Lab, University of Trento（特伦托大学信号与交互系统实验室） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

TabReason: A Reinforcement Learning-Enhanced Reasoning LLM for Explainable Tabular Data Prediction

Tommy Xu, Zhitian Zhang, Xiangyu Sun, Lauren Kelly Zung, Hossein Hajimirsadeghi, Greg Mori

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.LG

Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking

Chris Samarinas, Hamed Zamani

机构 * University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning

Pawin Taechoyotin, Daniel Acuna

机构 * University of Colorado Boulder（科罗拉多大学博尔德分校） ； Department of Information Science（信息科学系） ； ReviewerZero AI Inc.（ReviewerZero AI公司） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments 18 pages, 6 figures

Tabular Feature Discovery With Reasoning Type Exploration

Sungwon Han, Sungkyu Park, Seungeon Lee

机构 * GIST（韩国科学技术院） ； KDI School of Public Policy and Management（KDI公共政策与管理学院） ； KAIST（韩国科学技术院） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models

Mingyue Han, Yinglin Wang

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

Comments ACL2021, Main Conference, Short Paper

Journal ref Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Short Papers), pages 151-157, 2021

MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis

Yuting Zhang, Kaishen Yuan, Hao Lu, Yutao Yue, Jintai Chen, Kaishun Wu

机构 * The Hong Kong University of Science & Technology (Guangzhou)（香港科技大学（广州）） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

Standard Applicability Judgment and Cross-jurisdictional Reasoning: A RAG-based Framework for Medical Device Compliance

Yu Han, Aaron Ceross, Jeroen H. M. Bergmann

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

Abhay Gupta, Michael Lu, Kevin Zhu, Sean O'Brien, Vasu Sharma

机构 * Algoverse AI Research（Algoverse AI研究机构） ； University of California, Berkeley（加州大学伯克利分校） ； Meta FAIR Lab（Meta FAIR实验室） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

RadFabric: Agentic AI System with Reasoning Capability for Radiology

Wenting Chen, Yi Dong, Zhaojun Ding, Yucheng Shi, Yifan Zhou, Fang Zeng, Yijun Luo, Tianyu Lin, Yihang Su, Yichen Wu, Kai Zhang, Zhen Xiang, Tianming Liu, Ninghao Liu, Lichao Sun, Yixuan Yuan, Xiang Li

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

Comments 4 figures, 2 tables

CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs

Ai Jian, Weijie Qiu, Xiaokun Wang, Peiyu Wang, Yunzhuo Hao, Jiangbo Pei, Yichen Wei, Yi Peng, Xuchen Song

机构 * Skywork AI ； Kunlun Inc. 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments 36 pages

OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases

Yongrui Chen, Zhiqiang Liu, Jing Yu, Lin Ren, Nan Hu, Xinbang Dai, Jiajun Liu, Jiazhen Kang, Shenyu Zhang, Xinda Wang, Keyan Ding, Pengfei Shen, Haolei Zhu, Hongjie Deng, Yisong Wang, Tongtong Wu, Sheng Bi, Wen Zhang, Tianxing Wu, Qiu Ji, Haofen Wang, Wenliang Chen, Huajun Chen, Guilin Qi

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

Sanjoy Chowdhury, Mohamed Elmoghany, Yohan Abeysinghe, Junjie Fei, Sayan Nag, Salman Khan, Mohamed Elhoseiny, Dinesh Manocha

机构 * University of Maryland, College Park（马里兰大学College Park分校） ； KAUST（卡尔斯兰大学） ； MBZUAI（马克斯·普朗克人工智能研究所） ； University of Toronto（多伦多大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments Audio-visual learning, Audio-Visual RAG, Multi-Video Linkage

MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space

Anshul Singh, Chris Biemann, Jan Strich

机构 * Department of Information Technology, Panjab University（信息技术系，旁遮普大学） ； Language Technology Group, Universität Hamburg（语言技术组，汉堡大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics

Yaoming Zhu, Junxin Wang, Yiyang Li, Lin Qiu, ZongYu Wang, Jun Xu, Xuezhi Cao, Yuhuai Wei, Mingshi Wang, Xunliang Cai, Rong Ma

机构 * AGI-Eval ； Beijing Normal University（北京师范大学） ； Meituan（美团） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

WGSR-Bench: Wargame-based Game-theoretic Strategic Reasoning Benchmark for Large Language Models

Qiyue Yin, Pei Xu, Qiaozhe Li, Shengda Liu, Shengqi Shen, Tong Wang, Yihong Han, Xiaonan Zhao, Likun Yang, Shiyue Cao, Shiyu Qiu, Yuxuan Liu, Shizhao Yu, Lei Cui, Chengxin Yan, Jie Sun, Xiangquan Tang, Kaiqi Huang

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； University of Chinese Academy of Sciences（中国科学院大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments 15 pages, 17 figures

Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy

Sushant Gautam, Michael A. Riegler, Pål Halvorsen

机构 * Simula Metropolitan Center for Digital Engineering (SimulaMet)（Simula Metropolitan Center for Digital Engineering） ； Oslo Metropolitan University (OsloMet)（Oslo Metropolitan University） ； Simula Research Laboratory 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.LG

ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts

Ruiran Su, Jiasheng Si, Zhijiang Guo, Janet B. Pierrehumbert

机构 * University of Oxford（牛津大学） ； Qilu University of Technology(Shandong Academy of Sciences)（齐鲁工业大学（山东科学院）） ； Hong Kong University of Science and Technology（香港科学大学） ； Hong Kong University of Science and Technology (Guangzhou)（香港科学大学（广州）） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domains

Shijie Wang, Yilun Zhang, Zeyu Lai, Dexing Kong

机构 * School of Mathematical Sciences, Zhejiang University（浙江大学数学科学学院） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation

Ming Liu, Wensheng Zhang

机构 * Department of Computer Science（计算机科学系） ； Iowa State University（爱荷华州立大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain

Yunsoo Kim, Yusuf Abdulle, Honghan Wu

机构 * UCL（伦敦大学学院） ； King’s College London（伦敦国王学院） ； University of Glasgow（格拉斯哥大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

Towards Foundation Model on Temporal Knowledge Graph Reasoning

Jiaxin Pan, Mojtaba Nayyeri, Osama Mohammed, Daniel Hernandez, Rongchuan Zhang, Cheng Cheng, Steffen Staab

机构 * University of Stuttgart（斯图加特大学） ； University of Southampton（南安普顿大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach

Shuangyan Deng, Haizhou Peng, Jiachen Xu, Chunhou Liu, Ciprian Doru Giurcuaneanu, Jiamou Liu

机构 * University of Auckland（奥克兰大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning

Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, Yingjiu Li

机构 * Nanyang Technological University, Singapore（南洋理工大学） ； The Hong Kong University of Science（香港科学与技术大学） ； MetaTrust Labs（MetaTrust实验室） ； Singapore Management University, Singapore（新加坡管理学院） ； University of Oregon, Eugene, OR, USA（俄勒冈大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments This is a technical report by Nanyang Technological University. Updated to support Solidity, Java and C/C++

BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions

Saptarshi Sengupta, Shuhua Yang, Paul Kwong Yu, Fali Wang, Suhang Wang

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL

ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests

Shiyi Xu, Yiwen Hu, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Ji-Rong Wen

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL