大模型推理能力

Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media

Kun Li, Chenwei Dai, Wei Zhou, Songlin Hu

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

V-LoL: A Diagnostic Dataset for Visual Logical Learning

Lukas Helff, Wolfgang Stammer, Hikaru Shindo, Devendra Singh Dhami, Kristian Kersting

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI、cs.LG

Multi-Conditional Ranking with Large Language Models

Pouya Pezeshkpour, Estevam Hruschka

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.LG

ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents

Jakub Hoscilowicz, Bartosz Maj, Bartosz Kozakiewicz, Oleksii Tymoshchuk, Artur Janicki

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI、cs.LG

Comments The code for ClickAgent is available at github.com/Samsung/ClickAgent

The Future of Learning in the Age of Generative AI: Automated Question Generation and Assessment with Large Language Models

Subhankar Maity, Aniket Deroy

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments Book Chapter (Under Review)

NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding

Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, Yangqiu Song

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2024 findings. Dataset: https://github.com/HKUST-KnowComp/NegotiationToM

ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models

Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, Hayato Yamana

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutions

Laurène Vaugrante, Mathias Niepert, Thilo Hagendorff

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench

Karthik Valmeekam, Kaya Stechly, Subbarao Kambhampati

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Multi-Document Grounded Multi-Turn Synthetic Dialog Generation

Young-Suk Lee, Chulaka Gunasekara, Danish Contractor, Ramón Fernandez Astudillo, Radu Florian

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement

Simon Yu, Liangyu Chen, Sara Ahmadian, Marzieh Fadaee

专题命中 推理评测 ：reasoning(abstract)；math reasoning(abstract)；分类 cs.CL、cs.AI

Comments 21 pages, 6 figures

Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond

Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, Erik Cambria

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments 37 pages, 16 figures

Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness

Wenxuan Wang

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments PhD Thesis

SPICED: Syntactical Bug and Trojan Pattern Identification in A/MS Circuits using LLM-Enhanced Detection

Jayeeta Chaudhuri, Dhruv Thapar, Arjun Chaudhuri, Farshad Firouzi, Krishnendu Chakrabarty

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

Comments Accepted at PAINE'24

Fault Diagnosis in Power Grids with Large Language Model

Liu Jing, Amirul Rahman

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 11 pages

Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents

Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, Yejin Choi

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 2024 ICLR Spotlight. The dataset and code can be found at https://confaide.github.io

MolTC: Towards Molecular Relational Modeling In Language Models

Junfeng Fang, Shuai Zhang, Chang Wu, Zhengyi Yang, Zhiyuan Liu, Sihang Li, Kun Wang, Wenjie Du, Xiang Wang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.AI、cs.LG

Comments ACL 2024

On Limitation of Transformer for Learning HMMs

Jiachen Hu, Qinghua Liu, Chi Jin

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.AI、cs.LG

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ACL 2024. 24 pages. Project page: https://jykoh.com/vwa

UMass-BioNLP at MEDIQA-M3G 2024: DermPrompt -- A Systematic Exploration of Prompt Engineering with GPT-4V for Dermatological Diagnosis

Parth Vashisht, Abhilasha Lodha, Mukta Maddipatla, Zonghai Yao, Avijit Mitra, Zhichao Yang, Junda Wang, Sunjae Kwon, Hong Yu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments Accepted at NAACL-ClinicalNLP workshop 2024

The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews

Aleksi Huotala, Miikka Kuutila, Paul Ralph, Mika Mäntylä

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments Accepted to the International Conference on Evaluation and Assessment in Software Engineering (EASE), 2024 edition

Missed Connections: Lateral Thinking Puzzles for Large Language Models

Graham Todd, Tim Merino, Sam Earle, Julian Togelius

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 8 pages, 3 figures

MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation

Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, Dehai Min

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments This paper has been accepted as a long paper presentation by DASFAA 2024 Industrial Track

Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World

Guande Wu, Chen Zhao, Claudio Silva, He He

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model

Cheng Qian, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu

专题命中 推理评测 ：chain-of-thought(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments NAACL 2024 Main

SmartPlay: A Benchmark for LLMs as Intelligent Agents

Yue Wu, Xuan Tang, Tom M. Mitchell, Yuanzhi Li

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI、cs.LG

Balancing Exploration and Exploitation in LLM using Soft RLLF for Enhanced Negation Understanding

Ha-Thanh Nguyen, Ken Satoh

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments JURISIN 2024

Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models

Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wenpeng Yin, Lifu Huang

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments preprint, 15 pages

Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports

Qingqing Zhu, Xiuying Chen, Qiao Jin, Benjamin Hou, Tejas Sudharshan Mathai, Pritam Mukherjee, Xin Gao, Ronald M Summers, Zhiyong Lu

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL、cs.AI