大模型推理能力

Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contexts

Perapard Ngokpol, Kun Kerdthaisong, Pasin Buakhaw, Pitikorn Khlaisamniang, Supasate Vorathammathorn, Piyalitt Ittichaiwong, Nutchanon Yongsatianchot

机构 * Faculty of Engineering, Thammasat School of Engineering, Thammasat University（工程学院，泰国朱拉安高校工程学院，泰国朱拉安大学） ； Department of Computer Engineering and Digital Technology, Faculty of Engineering, Chulalongkorn University（计算机工程与数字技术系，工程学院，朱拉安大学） ； Artificial Intelligence Association of Thailand（泰国人工智能协会） ； School of Biomedical Engineering & Imaging Sciences, King’s College London（生物医学工程与成像科学学院，伦敦国王学院） ； Siriraj Informatics and Data Innovation Center (SIData+), Faculty of Medicine, Siriraj Hospital, Mahidol University（Siriraj信息与数据创新中心（SIData+），医学学院，Siriraj医院，玛希道大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking

Junhong Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai, Hua Wu

机构 * Baidu Inc.（百度公司） 纠错

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 11 pages, 6 figures, working in progress

DynaGuard: A Dynamic Guardian Model With User-Defined Policies

Monte Hoover, Vatsal Baherwani, Neel Jain, Khalid Saifullah, Joseph Vincent, Chirag Jain, Melissa Kazemi Rad, C. Bayan Bruss, Ashwinee Panda, Tom Goldstein

机构 * University of Maryland（马里兰大学） ； Capital One 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.LG

Comments 22 Pages

SoK: Measuring What Matters for Closed-Loop Security Agents

Mudita Khurana, Raunak Jain

机构 * IEEE 纠错

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports

Punit Kumar Singh, Nishant Kumar, Akash Ghosh, Kunal Pasad, Khushi Soni, Manisha Jaishwal, Sriparna Saha, Syukron Abu Ishaq Alfarozi, Asres Temam Abagissa, Kitsuchart Pasupa, Haiqin Yang, Jose G Moreno

机构 * Indian Institute of Technology Patna（印度理工学院帕纳布分校） ； Sardar Patel Institute of Technology（萨达尔·帕特尔技术学院） ； Universitas Gadjah Mada（加查马大学） ； King Mongkut’s Institute of Technology Ladkrabang（拉差班国王技术学院） ； Shenzhen Technology University（深圳技术大学） ； Université de Toulouse（图卢兹大学） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 52 pages, 56 figures; appearing at EMNLP'25

VideoScore2: Think before You Score in Generative Video Evaluation

Xuan He, Dongfu Jiang, Ping Nie, Minghao Liu, Zhengxuan Jiang, Mingyi Su, Wentao Ma, Junru Lin, Chun Ye, Yi Lu, Keming Wu, Benjamin Schneider, Quy Duc Do, Zhuofeng Li, Yiming Jia, Yuxuan Zhang, Guo Cheng, Haozhe Wang, Wangchunshu Zhou, Qunshu Lin, Yuanxing Zhang, Ge Zhang, Wenhao Huang, Wenhu Chen

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； University of Waterloo（滑铁卢大学） ； Independent（独立） ； M-A-P ； University of Toronto（多伦多大学） ； Zhejiang University（浙江大学） ； Abaka AI 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios

Haotian Luo, Huaisong Zhang, Xuelin Zhang, Haoyu Wang, Zeyu Qin, Wenjie Lu, Guozheng Ma, Haiying He, Yingsha Xie, Qiyang Zhou, Zixuan Hu, Hongze Mi, Yibo Wang, Naiqiang Tan, Hong Chen, Yi R. Fung, Chun Yuan, Li Shen

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge

Jinghan Peng, Jingwen Wang, Xing Yu, Dehui Du

机构 * East China Normal University（东华师范大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection

Ira Ceka, Feitong Qiao, Anik Dey, Aastha Valecha, Gail Kaiser, Baishakhi Ray

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samples

Qixin Sun, Ziqin Wang, Hengyuan Zhao, Yilin Li, Kaiyou Song, Linjiang Huang, Xiaolin Hu, Qingpei Guo, Si Liu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

CANDY: Benchmarking LLMs' Limitations and Assistive Potential in Chinese Misinformation Fact-Checking

Ruiling Guo, Xinwei Yang, Chen Huang, Tong Zhang, Yong Hu

机构 * School of Cyber Science and Engineering, Sichuan University, China（四川大学网络科学与工程学院） ； College of Computer Science, Sichuan University, China（四川大学计算机学院） ； Institute of Data Science, National University of Singapore, Singapore（新加坡国立大学数据科学研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments Findings of EMNLP 2025

ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents

Tianjian Liu, Fanqi Wan, Jiajian Guo, Xiaojun Quan

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments 21 pages, 6 Figures

RubikSQL: Lifelong Learning Agentic Knowledge Base as an Industrial NL2SQL System

Zui Chen, Han Li, Xinhao Zhang, Xiaoyu Chen, Chunyin Dong, Yifeng Wang, Xin Cai, Su Zhang, Ziqi Li, Chi Ding, Jinxu Li, Shuai Wang, Dousheng Zhao, Sanhai Gao, Guangyi Liu

机构 * Huawei Company（华为公司） ； Cornell University（康奈尔大学） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 3 figures, 3 tables, to be submitted to VLDB 2026 (PVLDB Volume 19)

ImF: Implicit Fingerprint for Large Language Models

Jiaxuan Wu, Wanli Peng, Hang Fu, Yiming Xue, Juan Wen

机构 * China Agricultural University（中国农业大学） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 13 pages, 6 figures

Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?

Henrique Godoy

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation

Yichi Zhang, Yao Huang, Yifan Wang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Huanran Chen, Xiao Yang, Xingxing Wei, Hang Su, Yinpeng Dong, Jun Zhu

机构 * Department of Computer Science and Technology, College of AI, Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University（计算机科学与技术系、人工智能学院、人工智能研究所、清华-博世联合机器学习中心、THBI实验室、BNRist中心、清华大学） ； Institute of Artificial Intelligence, Beihang University（人工智能研究院、北航） ； RealAI 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments For Appendix, please refer to arXiv:2406.07057

EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting

Guanrou Yang, Chen Yang, Qian Chen, Ziyang Ma, Wenxi Chen, Wen Wang, Tianrui Wang, Yifan Yang, Zhikang Niu, Wenrui Liu, Fan Yu, Zhihao Du, Zhifu Gao, ShiLiang Zhang, Xie Chen

机构 * Shanghai Jiao Tong University（上海交通大学） ； Tongyi Speech Lab（通义语音实验室） ； Tianjin University（天津大学） ； Zhejiang University（浙江大学） ； Shanghai Jiao Tong University, Shanghai Innovation Institute（上海交通大学上海创新研究院） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ACMMM 2025

Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performance

Wael Etaiwi, Bushra Alhijawi

机构 * Princess Sumaya University for Technology（普里incess苏玛亚技术大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution

Falaah Arif Khan, Nivedha Sivakumar, Yinong Oliver Wang, Katherine Metcalf, Cezanne Camacho, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff

机构 * Apple（苹果公司） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models

Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, Bryan Catanzaro

机构 * NVIDIA, USA（NVIDIA公司） ； University of Maryland, College Park, USA（马里兰大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments Code, Datasets, and Models: https://research.nvidia.com/labs/adlr/AF3/ ; Updates in v2: Updated results for new thinking mode ckpts, added qualitative figure, added note on fully open claim, add email ID for corresponding authors

Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluation

Ashray Gupta, Rohan Joseph, Sunny Rai

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Affective-ROPTester: Capability and Bias Analysis of LLMs in Predicting Retinopathy of Prematurity

Shuai Zhao, Yulin Zhang, Luwei Xiao, Xinyi Wu, Yanhao Jia, Zhongliang Guo, Xiaobao Wu, Cong-Duy Nguyen, Guoming Zhang, Anh Tuan Luu

机构 * College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院） ； Department of Ophthalmology, Huizhou First Hospital（惠州市第一人民医院眼科部） ； school of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院） ； School of Education, Shanghai Jiao Tong University（上海交通大学教育学院） ； School of Computer Science, University of St Andrews（圣安德鲁大学计算机科学学院） ； Shenzhen Eye Hospital, Shenzhen Eye Medical Center, Southern Medical University（深圳眼医院、深圳眼科医院、南方医科大学） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Test-time Scaling Techniques in Theoretical Physics -- A Comparison of Methods on the TPBench Dataset

Zhiqi Gao, Tianyi Li, Yurii Kvasiuk, Sai Chaitanya Tadepalli, Maja Rudolph, Daniel J. H. Chung, Frederic Sala, Moritz Münchmeyer

机构 * Department of Physics, University of Wisconsin-Madison（威斯康星大学麦迪逊分校物理系） ； Department of Computer Science, University of Wisconsin-Madison（威斯康星大学麦迪逊分校计算机科学系） ； Data Science Institute (DSI), University of Wisconsin-Madison（威斯康星大学麦迪逊分校数据科学研究院） ； Department of Physics, Indiana University, Bloomington（印第安纳大学布卢明顿分校物理系） 纠错

专题命中 推理评测 ：reasoning(abstract)；verifier(abstract)；分类 cs.AI、cs.LG

Comments 23 pages, 6 figures

LOGICPO: Efficient Translation of NL-based Logical Problems to FOL using LLMs and Preference Optimization

Koushik Viswanadha, Deepanway Ghosal, Somak Aditya

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI、cs.LG

Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity

Hangfan Zhang, Zhiyao Cui, Jianhao Chen, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, Shuyue Hu

机构 * Pennsylvania State University（宾夕法尼亚州立大学） ； Northwestern Polytechnical University（西北工业大学） ； Singapore Management University（新加坡管理学院） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Nanjing University（南京大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.LG

Comments This position paper takes a critical view of the status quo of MAD research, and outline multiple potential directions to improve MAD

Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack

Murong Yue, Ziyu Yao

机构 * George Mason University（乔治·马歇尔大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ACL Findings, 2025

PsychBench: A comprehensive and professional benchmark for evaluating the performance of LLM-assisted psychiatric clinical practice

Shuyu Liu, Ruoxi Wang, Ling Zhang, Xuequan Zhu, Rui Yang, Xinzhu Zhou, Fei Wu, Zhi Yang, Cheng Jin, Gang Wang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning

Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Jie Xie, Wei Zhou, Wang Xu, Yuanheng Zhang, Zhou Su, Zhongwu Zhai, Xiaoming Liu, Yudong Mei, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments Updated results in Table 2 and Table 3; The project is available at https://github.com/OpenBMB/AgentCPM-GUI

Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model

Xue Wang, Tian Zhou, Jinyang Gao, Bolin Ding, Jingren Zhou

机构 * Alibaba-inc（阿里巴巴集团） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving

Vincenzo Colle, Mohamed Sana, Nicola Piovesan, Antonio De Domenico, Fadhel Ayed, Merouane Debbah

机构 * Paris Research Center, Huawei Technologies（巴黎研究中心，华为技术） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments 6 pages