大模型推理能力

WebQuest: A Benchmark for Multimodal QA on Web Page Sequences

Maria Wang, Srinivas Sunkara, Gilles Baechler, Jason Lin, Yun Zhu, Fedir Zubach, Lei Shu, Jindong Chen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Can we only use guideline instead of shot in prompt?

Jiaxiang Chen, Song Wang, Zhucong Li, Wayne Xiong, Lizhen Qu, Zenglin Xu, Yuan Qi

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.AI

CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization

Ziwei Gong, Lin Ai, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Zehui Wu, Ahmad Emami, Julia Hirschberg

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, Zack Hui

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

MAGDA: Multi-agent guideline-driven diagnostic assistance

David Bani-Harouni, Nassir Navab, Matthias Keicher

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models

Xiyuan Zhou, Huan Zhao, Yuheng Cheng, Yuji Cao, Gaoqi Liang, Guolong Liu, Wenxuan Liu, Yan Xu, Junhua Zhao

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, Rifat Shahriyar

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection

Peter Røysland Aarnes, Vinay Setty, Petra Galuščáková

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted to CLEF2024 CheckThat!

Revisiting Relation Extraction in the era of Large Language Models

Somin Wadhwa, Silvio Amir, Byron C. Wallace

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments Accepted to ACL 2023

Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping

Wenhao Zhu, Sizhe Liu, Shujian Huang, Shuaijie She, Chris Wendler, Jiajun Chen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Towards Temporal Change Explanations from Bi-Temporal Satellite Images

Ryo Tsujimoto, Hiroki Ouchi, Hidetaka Kamigaito, Taro Watanabe

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments 11 pages, 6 figures

Leveraging large language models for nano synthesis mechanism explanation: solid foundations or mere conjectures?

Yingming Pu, Liping Huang, Tao Lin, Hongyu Chen

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.LG

Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models through Question Answering from Text to Video

Zhengbang Yang, Haotian Xia, Jingxi Li, Zezhi Chen, Zhuangdi Zhu, Weining Shen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models

Qiyao Wang, Jianguo Huang, Shule Lu, Yuan Lin, Kan Xu, Liang Yang, Hongfei Lin

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach

Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, Jun Wang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

LogiCode: an LLM-Driven Framework for Logical Anomaly Detection

Yiheng Zhang, Yunkang Cao, Xiaohao Xu, Weiming Shen

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.LG

A Survey of Useful LLM Evaluation

Ji-Lun Peng, Sijia Cheng, Egil Diau, Yung-Yu Shih, Po-Heng Chen, Yen-Ting Lin, Yun-Nung Chen

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments Accepted to ACL 2024. Camera Ready. Our dataset is available at https://github.com/zt991211/CLAMBER

WorldAfford: Affordance Grounding based on Natural Language Instructions

Changmao Chen, Yuren Cong, Zhen Kan

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Evaluating and Modeling Social Intelligence: A Comparative Study of Human and AI Capabilities

Junqi Wang, Chunhui Zhang, Jiapeng Li, Yuxi Ma, Lixing Niu, Jiaheng Han, Yujia Peng, Yixin Zhu, Lifeng Fan

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments Also published in Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci), 2024

SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications

Liang Xu, Lei Zhu, Yaotong Wu, Hang Xue

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments 11 pages, 19 figures, and tables

Language Models Still Struggle to Zero-shot Reason about Time Series

Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, Tim Althoff

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Measuring Taiwanese Mandarin Language Understanding

Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, Yun-Nung Chen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Preprint. Under review

Can a GPT4-Powered AI Agent Be a Good Enough Performance Attribution Analyst?

Bruno de Melo, Jamiel Sheikh

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.AI

HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models

Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jaecheol Lee, Je Won Yeom, Jihyu Jung, Jung Woo Kim, Songseong Kim

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments Accepted at LREC-COLING 2024

Metaphor Understanding Challenge Dataset for LLMs

Xiaoyu Tong, Rochelle Choenni, Martha Lewis, Ekaterina Shutova

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles

Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang, Wuhe Zou, Weidong Zhang, Hai-Tao Zheng

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments Accepted by LREC-COLING 2024

Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games

Dekun Wu, Haochen Shi, Zhiyuan Sun, Bang Liu

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments

Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, Lijie Wen

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models

Ayo Adedeji, Sarita Joshi, Brendan Doohan

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments 31 pages, 17 figures