大模型推理能力

CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedback

En-Qi Tseng, Pei-Cing Huang, Chan Hsu, Peng-Yi Wu, Chan-Tung Ku, Yihuang Kang

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.AI

Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents

Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Caiming Xiong, Shiva Kumar Pentyala, Chien-Sheng Wu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models

Zhongzhan Huang, Shanshan Zhong, Pan Zhou, Shanghua Gao, Marinka Zitnik, Liang Lin

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

Comments Accepted by TPAMI. arXiv admin note: text overlap with arXiv:2312.02439

How to Get Your LLM to Generate Challenging Problems for Evaluation

Arkil Patel, Siva Reddy, Dzmitry Bahdanau

专题命中 推理评测 ：reasoning(abstract)；math reasoning(abstract)；分类 cs.CL

Benchmarking Language Model Creativity: A Case Study on Code Generation

Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, Sanjeev Khudanpur, Meng Jiang, Daniel Khashabi

专题命中 推理评测 ：reasoning(abstract)；self-correction(abstract)；分类 cs.CL

ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

Haiyang Shen, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments ICLR'25: https://openreview.net/forum?id=kKILfPkhSz

OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models

Chongren Sun, Yuran Li, Di Wu, Benoit Boulet

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans

Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, Raymond Mooney

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2024 Main Conference

OpenLKA: an open dataset of lane keeping assist from market autonomous vehicles

Yuhang Wang, Abdulaziz Alhuraish, Shengming Yuan, Shuyi Wang, Hao Zhou

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.LG

SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity

Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, Xiapu Luo

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension

Yanbo Fang, Ruixiang Tang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

How susceptible are LLMs to Logical Fallacies?

Amirreza Payandeh, Dan Pluth, Jordan Hosier, Xuesu Xiao, Vijay K. Gurbani

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Journal ref https://aclanthology.org/2024.lrec-main.726/

Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios

Zhi Chen, Lingxiao Jiang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments Paper accepted to the SANER 2025 Conference Research Track

From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering

Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Sam Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations

Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Hyeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, Edward Choi

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models

Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, Liwen Zhang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data

Junhao Liu, Siwei Xu, Lei Zhang, Jing Zhang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Leveraging Large Language Models for Comparative Literature Summarization with Reflective Incremental Mechanisms

Fernando Gabriela Garcia, Spencer Burns, Harrison Fuller

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

Comments 8 pages

Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos

Zhimin Shao, Jialang Xu, Danail Stoyanov, Evangelos B. Mazomenos, Yueming Jin

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Comments 8 pages, 4 figures

Journal ref IEEE Robotics and Automation Letters, vol. 9, no. 12, pp. 11513-11520, Dec. 2024

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, Jiashi Feng

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

Comments EMNLP 2024

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, Wenhu Chen

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

Comments This version has been accepted and published at NeurIPS 2024 Track Datasets and Benchmarks (Spotlight)

MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering

Yizhen Luo, Zikun Nie, Massimo Hong, Suyuan Zhao, Hao Zhou, Zaiqing Nie

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.LG

Comments NeurIPS 2024 poster

Rationale-Aware Answer Verification by Pairwise Self-Evaluation

Akira Kawabata, Saku Sugawara

专题命中 推理评测 ：reasoning(abstract)；verifier(abstract)；分类 cs.CL

Comments EMNLP 2024

Are LLMs Good Zero-Shot Fallacy Classifiers?

Fengjun Pan, Xiaobao Wu, Zongrui Li, Anh Tuan Luu

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted to EMNLP2024 main conference

Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study

Gonzalo Martínez, José Alberto Hernández, Javier Conde, Pedro Reviriego, Elena Merino

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Journal ref ACM Transactions on Intelligent Systems and Technology, 2024

Pragmatic Competence Evaluation of Large Language Models for the Korean Language

Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, Sungeun Lee

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments 38th Pacific Asia Conference on Language, Information and Computation

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning

Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments EMNLP 2024

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, Hinrich Schütze

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments EMNLP 2024 - Findings

NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian

Peng Liu, Lemei Zhang, Terje Farup, Even W. Lauvrak, Jon Espen Ingvaldsen, Simen Eide, Jon Atle Gulla, Zhirong Yang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted at EMNLP 2024 Main Conference. Code available at https://github.com/Smartmedia-AI/NorGLM/

Task-oriented Prompt Enhancement via Script Generation

Chung-Yu Wang, Alireza DaghighFarsoodeh, Hung Viet Pham

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Comments 17 pages + reference