大模型推理能力

UQ: Assessing Language Models on Unsolved Questions

Fan Nie, Ken Ziyu Liu, Zihao Wang, Rui Sun, Wei Liu, Weijia Shi, Huaxiu Yao, Linjun Zhang, Andrew Y. Ng, James Zou, Sanmi Koyejo, Yejin Choi, Percy Liang, Niklas Muennighoff

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments FN, KZL, and NM are project co-leads and contributed equally. Project website: https://uq.stanford.edu

A Universal Framework for Offline Serendipity Evaluation in Recommender Systems via Large Language Models

Yu Tokutake, Kazushi Okamoto, Kei Harada, Atsushi Shibata, Koki Karube

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

Journal ref The 34th ACM International Conference on Information and Knowledge Management (CIKM 2025)

Self-Supervised Prompt Optimization

Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Xinbing Liang, Fengwei Teng, Jinhao Tu, Fashen Ren, Xiangru Tang, Sirui Hong, Chenglin Wu, Yuyu Luo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine

Maxime Griot, Jean Vanderdonckt, Demet Yuksel, Coralie Hemptinne

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ACL 2025 main

ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine

Junying Chen, Zhenyang Cai, Zhiheng Liu, Yunjin Yang, Rongsheng Wang, Qingying Xiao, Xiangyi Feng, Zhan Su, Jing Guo, Xiang Wan, Guangjun Yu, Haizhou Li, Benyou Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples

Haiquan Hu, Jiazhi Jiang, Shiyou Xu, Ruhan Zeng, Tian Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Submit to AAAI 2026

LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests

Zachariah Sollenberger, Rahul Patel, Saieda Ali Zada, Sunita Chandrasekaran

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)

Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

Hongchen Wang, Kangming Li, Scott Ramsay, Yao Fehlis, Edward Kim, Jason Hattrick-Simpers

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

OrderChain: Towards General Instruct-Tuning for Stimulating the Ordinal Understanding Ability of MLLM

Jinhong Wang, Shuo Tong, Jian liu, Dongqi Tang, Weiqiang Wang, Wentong Li, Hongxia Xu, Danny Chen, Jintai Chen, Jian Wu

机构 * College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院） ； Transvascular Implantation Devices Research Institute and Liangzhu Laboratory（血管植入物研究机构和良渚实验室） ； Ant Group（蚂蚁集团） ； University of Notre Dame（圣母大学） ； HKUST (Guangzhou)（香港科技大学（广州）） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)

Comments Accepted by ICCV 2025

Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?

Yang Yao, Lingyu Li, Jiaxin Song, Chiyu Chen, Zhenqi He, Yixu Wang, Xin Wang, Tianle Gu, Jie Li, Yan Teng, Yingchun Wang

机构 * Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； The University of Hong Kong（香港大学） ； Shanghai Jiao Tong University（上海交通大学） ； The Hong Kong University of Science and Technology（香港科学与技术大学） ； Fudan University（复旦大学） ； Tsinghua University（清华大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy

Alexander Duffy, Samuel J Paech, Ishana Shastri, Elizabeth Karpinski, Baptiste Alloui-Cros, Tyler Marques, Matthew Lyle Olson

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

MultiMedEdit: A Scenario-Aware Benchmark for Evaluating Knowledge Editing in Medical VQA

Shengtao Wen, Haodong Chen, Yadong Wang, Zhongying Pan, Xiang Chen, Yu Tian, Bo Qian, Dong Liang, Sheng-Jun Huang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Under Review

VL-MedGuide: A Visual-Linguistic Large Model for Intelligent and Explainable Skin Disease Auxiliary Diagnosis

Kexin Yu, Zihan Xu, Jialei Xie, Carter Adams

机构 * Jiangsu Ocean University（江苏海洋大学） ； Federal University of Bahia（巴伊亚联邦大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports

Jin Khye Tan, En Jun Choong, Ethan Jeremiah Chitty, Yan Pheng Choo, John Hsin Yang Wong, Chern Eu Cheah

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 14 figures, 5 tables. Evaluation code (LLM-as-a-judge and Markdown TEDS) is available at https://github.com/jinkhye/MyFinMarkdown. The development dataset and evaluation benchmark are available on Hugging Face at https://huggingface.co/datasets/jinkhye/MyFinMarkdown-sample and https://huggingface.co/datasets/jinkhye/MyFinMarkdown-bench respectively

CleanUpBench: Embodied Sweeping and Grasping Benchmark

Wenbo Li, Guanting Chen, Tao Zhao, Jiyao Wang, Tianxin Hu, Yuwen Liao, Weixiang Guo, Shenghai Yuan

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)

ArXivBench: When You Should Avoid Using ChatGPT for Academic Writing

Ning Li, Jingran Zhang, Justin Cui

机构 * University of California, Los Angeles（加州大学洛杉矶分校） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation

Radhika Dua, Young Joon, Kwon, Siddhant Dogra, Daniel Freedman, Diana Ruan, Motaz Nashawaty, Danielle Rigau, Daniel Alexander Alber, Kang Zhang, Kyunghyun Cho, Eric Karl Oermann

机构 * New York University（纽约大学） ； NYU Langone Health（纽约大学兰格医学中心） ； Genentech（基因泰克） ； NYU Grossman School of Medicine（纽约大学格罗斯曼医学院） ； Wenzhou Medical University（温州医科大学） ； Macau University of Science and Technology（澳门科学大学） 纠错

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI、cs.LG

Diagnostic Accuracy of Open-Source Vision-Language Models on Diverse Medical Imaging Tasks

Gustav Müller-Franzes, Debora Jutz, Jakob Nikolas Kather, Christiane Kuhl, Sven Nebelung, Daniel Truhn

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

Libra: Leveraging Temporal Images for Biomedical Radiology Analysis

Xi Zhang, Zaiqiao Meng, Jake Lever, Edmond S. L. Ho

机构 * Information Retrieval Group（信息检索组） ； AI4BioMed Lab（AI4BioMed实验室） ； School of Computing Science（计算科学学院） ； University of Glasgow（格拉斯哥大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 30 pages, 5 figures, Adding Appendix

Journal ref Association for Computational Linguistics, 2025

Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks

Jianghui Wang, Vinay Joshi, Saptarshi Majumder, Xu Chao, Bin Ding, Ziqiong Liu, Pratik Prabhanjan Brahma, Dong Li, Zicheng Liu, Emad Barsoum

机构 * Advanced Micro Devices, Inc.（先进微器件公司） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

VisNumBench: Evaluating Number Sense of Multimodal Large Language Models

Tengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming

机构 * College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院） ； Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济发展实验室） ； Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际 Graduate School） ； Shenzhen Technology University（深圳技术大学） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)

Comments accepted by ICCV 2025

UserBench: An Interactive Gym Environment for User-Centric Agents

Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, Huan Wang

机构 * Salesforce AI Research（Salesforce AI研究部） ； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 25 Pages, 17 Figures, 6 Tables

Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning

Shengyuan Wang, Jie Feng, Tianhui Liu, Dan Pei, Yong Li

机构 * College of AI, Tsinghua University（清华大学人工智能学院） ； Department of Electronic Engineering, BNRist, Tsinghua University（清华大学电子工程系） ； School of Electronic and Information Engineering, Beijing Jiaotong University（北京交通大学电子信息学院） ； Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 19 pages, 9 figures

Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts

Sang-Woo Lee, Sohee Yang, Donghyun Kwak, Noah Y. Siegel

机构 * Google Deepmind（谷歌DeepMind） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

SegSub: Evaluating Robustness to Knowledge Conflicts and Hallucinations in Vision-Language Models

Peter Carragher, Nikitha Rao, Abhinand Jha, R Raghav, Kathleen M. Carley

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Journal ref MisD 2025: 1st Workshop on Misinformation Detection in the Era of LLMs

Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models

Junjie Wu, Tsz Ting Chung, Kai Chen, Dit-Yan Yeung

机构 * The Hong Kong University of Science and Technology（香港科学与技术大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by TMLR 2025. Project Page: https://kaichen1998.github.io/projects/tri-he/

Evaluating Generated Commit Messages with Large Language Models

Qunhong Zeng, Yuxia Zhang, Zexiong Ma, Bo Jiang, Ningyuan Sun, Klaas-Jan Stol, Xingyu Mou, Hui Liu

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding

Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, Radha Poovendran

机构 * Microsoft GenAI（微软生成人工智能） ； University of Washington（华盛顿大学） ； The University of Texas at Austin（德克萨斯大学奥斯汀分校） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ACL 2025. Codes and Data: https://kodcode-ai.github.io/

Entity Re-identification in Visual Storytelling via Contrastive Reinforcement Learning

Daniel A. P. Oliveira, David Martins de Matos

机构 * INESC-ID（INESC-ID研究所） ； Instituto Superior Técnico, Universidade de Lisboa（里斯本大学技术学院，里斯本大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

Comments 7 pages

MAEBE: Multi-Agent Emergent Behavior Framework

Sinem Erisken, Timothy Gothard, Martin Leitgab, Ram Potham

机构 * Independent Researcher（独立研究者） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint. This work has been submitted to the Multi-Agent Systems Workshop at ICML 2025 for review