大模型推理能力

StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation

Shenyang Liu, Yang Gao, Shaoyan Zhai, Liqiang Wang

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 2024 IEEE International Conference on Big Data (BigData)

ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis

Xu Wang, Jiaju Kang, Puyu Han, Yubao Zhao, Qian Liu, Liwenfei He, Lingqiong Zhang, Lingyun Dai, Yongcheng Wang, Jie Tao

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges

Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, Guangyao Shi

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 22 pages, 3 figures

Journal ref Navigating the Future: Ensuring Trustworthiness in Multi-Modal Open-World Intelligence @ CVPR, 2025

JudgeBench: A Benchmark for Evaluating LLM-based Judges

Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published as a conference paper at ICLR 2025

DAgent: A Relational Database-Driven Data Analysis Report Generation Agent

Wenyi Xu, Yuren Mao, Xiaolu Zhang, Chao Zhang, Xuemei Dong, Mengfei Zhang, Yunjun Gao

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)

StepGrade: Grading Programming Assignments with Context-Aware LLMs

Mohammad Akyash, Kimia Zamiri Azar, Hadi Mardani Kamali

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)

Comments Accepted to the 15th IEEE Integrated STEM Education Conference (ISEC)

VisionArena: 230K Real World User-VLM Conversations with Preference Labels

Christopher Chou, Lisa Dunlap, Koki Mashita, Krishna Mandal, Trevor Darrell, Ion Stoica, Joseph E. Gonzalez, Wei-Lin Chiang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)

Comments updated for CVPR Camera Ready

Survey on Evaluation of LLM-based Agents

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI、cs.LG

Reliable and Efficient Amortized Model-based Evaluation

Sang Truong, Yuheng Tu, Percy Liang, Bo Li, Sanmi Koyejo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving

Tin Stribor Sohn, Philipp Reis, Maximilian Dillitzer, Johannes Bach, Jason J. Corso, Eric Sax

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)

Comments Submitted to IEEE IAVVC 2025, Under Review

Emergent Abilities in Large Language Models: A Survey

Leonardo Berti, Flavio Giorgi, Gjergji Kasneci

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning

Jiazheng Liu, Sipeng Zheng, Börje F. Karlsson, Zongqing Lu

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

Towards Conversational AI for Disease Management

Anil Palepu, Valentin Liévin, Wei-Hung Weng, Khaled Saab, David Stutz, Yong Cheng, Kavita Kulkarni, S. Sara Mahdavi, Joëlle Barral, Dale R. Webster, Katherine Chou, Avinatan Hassidim, Yossi Matias, James Manyika, Ryutaro Tanno, Vivek Natarajan, Adam Rodman, Tao Tu, Alan Karthikesalingam, Mike Schaekermann

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 62 pages, 7 figures in main text, 36 figures in appendix

Beyond Next Word Prediction: Developing Comprehensive Evaluation Frameworks for measuring LLM performance on real world applications

Vishakha Agrawal, Archie Chaudhury, Shreya Agrawal

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Learning from Failures in Multi-Attempt Reinforcement Learning

Stephen Chung, Wenyu Du, Jie Fu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments preprint

Bias Unveiled: Investigating Social Bias in LLM-Generated Code

Lin Ling, Fazle Rabbi, Song Wang, Jinqiu Yang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)

Comments accepted for publication in the Association for the Advancement of Artificial Intelligence (AAAI), 2025

TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records

Hejie Cui, Alyssa Unell, Bowen Chen, Jason Alan Fries, Emily Alsentzer, Sanmi Koyejo, Nigam Shah

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint

CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, Weihao Zeng, Yejie Wang, Zhuoma GongQue, Jianing Yu, Qiuna Tan, Weiran Xu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ICLR 2025

Can Generative AI Support Patients' & Caregivers' Informational Needs? Towards Task-Centric Evaluation Of AI Systems

Shreya Rajagopal, Jae Ho Sohn, Hari Subramonyam, Shiwali Mohan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Journal ref Joint Proceedings of the ACM IUI Workshops 2025, March 24-27, 2025, Cagliari, Italy

Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs

Kuan Lok Zhou, Jiayi Chen, Siddharth Suresh, Reuben Narad, Timothy T. Rogers, Lalit K Jain, Robert D Nowak, Bob Mankoff, Jifan Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Small Language Models: Survey, Measurements, and Insights

Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, Mengwei Xu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Multilingual LLMs Inherently Reward In-Language Time-Sensitive Semantic Alignment for Low-Resource Languages

Ashutosh Bajpai, Tanmoy Chakraborty

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence

Bhavik Agarwal, Ishan Joshi, Viktoria Rojkova

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Reward Models Identify Consistency, Not Causality

Yuhui Xu, Hanze Dong, Lei Wang, Caiming Xiong, Junnan Li

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 16 pages

Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models

James Fodor

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 10 pages

LIMR: Less is More for RL Scaling

Xuefeng Li, Haoyang Zou, Pengfei Liu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 6pages

Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers

Aivin V. Solatorio, Rafael Macalaba, James Liounis

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Project GitHub repository at https://github.com/worldbank/ai4data-use

Evaluating the Performance of Large Language Models via Debates

Behrad Moniri, Hamed Hassani, Edgar Dobriban

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

ACCORD: Closing the Commonsense Measurability Gap

François Roewer-Després, Jinyue Feng, Zining Zhu, Frank Rudzicz

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments For leaderboard and dataset download, see https://www.codabench.org/competitions/3160/ For source code, see https://github.com/francois-rd/accord/

ALMANACS: A Simulatability Benchmark for Language Model Explainability

Edmund Mills, Shiye Su, Stuart Russell, Scott Emmons

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code is available at https://github.com/edmundmills/ALMANACS}{https://github.com/edmundmills/ALMANACS