语言大模型 / LLM

Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Models

Zongxia Li, Lorena Calvo-Bartolomé, Alexander Hoyle, Paiheng Xu, Alden Dima, Juan Francisco Fung, Jordan Boyd-Graber

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments 22 Pages. LLM for Data Exploration and content analysis, Topic Models. 63rd Annual Meeting of the Association for Computational Linguistics (2025)

SIMCOPILOT: Evaluating Large Language Models for Copilot-Style Code Generation

Mingchao Jiang, Abhinav Jain, Sophia Zorek, Chris Jermaine

机构 * Department of Computer Science（计算机科学系） ； Department of Statistics（统计学系） ； Rice University（里士满大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.LG

Comments Keywords: Benchmark Dataset, LLM Evaluation, Gen-AI, Program Synthesis; TLDR: SimCoPilot is a benchmark for evaluating LLMs as "copilot"-style interactive coding assistants, testing their ability to integrate and complete code within complex real-world software environments

Large Language Models Penetration in Scholarly Writing and Peer Review

Li Zhou, Ruijie Zhang, Xunlian Dai, Daniel Hershcovich, Haizhou Li

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments Transparency in NLP, LLM-generated text evaluation and detection, LLM Penetration, Scholarly Credibility and Accountability

Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias

Jayanta Sadhu, Maneesha Rani Saha, Rifat Shahriyar

专题命中 评测与基准 ：language model(title,abstract)；large language model(title,abstract)；LLM(abstract)；分类 cs.CL

Comments Accepted at The First Workshop on Language Models for Low-Resource Languages (LoResLM) at COLING 2025

Can Large Language Models Identify Authorship?

Baixiang Huang, Canyu Chen, Kai Shu

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments Accepted to EMNLP 2024 Findings. The main paper is 9 pages long, with 16 pages total. The code, results, dataset, and additional resources are available on the project website: https://llm-authorship.github.io/

Measuring the Inconsistency of Large Language Models in Preferential Ranking

Xiutian Zhao, Ke Wang, Wei Peng

专题命中 评测与基准 ：language model(title,abstract)；large language model(title,abstract)；LLM(abstract)；分类 cs.CL

Comments In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024)

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

Samuel Marks, Max Tegmark

专题命中 评测与基准 ：language model(title,abstract)；large language model(title,abstract)；LLM(abstract)；分类 cs.AI

Comments Conference on Language Modeling, 2024

Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectives

Azmine Toushik Wasi, Raima Islam, Mst Rafia Islam, Taki Hasan Rafi, Dong-Kyu Chae

专题命中 评测与基准 ：language model(title,abstract)；large language model(title,abstract)；LLM(abstract)；分类 cs.CL

Comments 10 Pages, 4 Figures. Accepted to the 1st Human-centered Evaluation and Auditing of Language Models Workshop at CHI 2024 (Workshop website: https://heal-workshop.github.io/#:~:text=Exploring%20Bengali%20Religious%20Dialect%20Biases%20in%20Large%20Language%20Models%20with%20Evaluation%20Perspectives)

Topics, Authors, and Institutions in Large Language Model Research: Trends from 17K arXiv Papers

Rajiv Movva, Sidhika Balachandar, Kenny Peng, Gabriel Agostini, Nikhil Garg, Emma Pierson

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments NAACL 2024. Data & code available at https://github.com/rmovva/LLM-publication-patterns-public

Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages

Jakub Hoscilowicz, Pawel Pawlowski, Marcin Skorupa, Marcin Sowański, Artur Janicki

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments Code and info on model checkpoint are available at https://github.com/Samsung/MT-LLM-NLU

PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion

Zekai Zhang, Yiduo Guo, Yaobo Liang, Dongyan Zhao, Nan Duan

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments LLM evaluation, Multi-turn, Multi-language, Multi-modal benchmark

PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion

Yiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao, Nan Duan

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments LLM evaluation, PPT task completion

A Closer Look into Automatic Evaluation Using Large Language Models

Cheng-Han Chiang, Hung-yi Lee

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；LLM(abstract,comments)；分类 cs.CL

Comments EMNLP 2023 findings (short paper). Code: https://github.com/d223302/A-Closer-Look-To-LLM-Evaluation/

Bayesian Partner Modelling enables Adaptive Replanning for LLM Coordination

贝叶斯伙伴建模支持大语言模型协同的自适应重规划

Harsh Goel, Aditya Sai Ellendula, Vaishnav Tadiparthi, Ehsan Moradi Pari, Hossein Nourkhiz Mahjoub, Sandeep P. Chinchali

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 该研究针对LLM多智能体协作中伙伴策略变化导致的重规划问题，提出BayesBeliefAgent方法，在Overcooked环境中显著缩小信念-行动差距并减少重规划次数。

Comments Preprint

Procedural Collapse: A Structural Account of Disengagement in LLM-Assisted Writing

程序崩溃：大语言模型辅助写作中脱离的结构解释

JaeWon Kim, Katelyn X. Mei

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 本文针对学生使用LLM写作时脱离的问题，提出结构层面解释，指出当前界面引发程序崩溃，给出分解式交互等设计方向以支持AI辅助写作。

LLM-Guided Evolutionary Search for Algebraic T-Count Optimization

基于LLM的引导进化搜索在代数T计数优化中的应用

Daniil Fisher, Valentin Khrulkov, Mikhail Saygin, Ivan Oseledets, Stanislav Straupe

专题命中 评测与基准 ：LLM(title,title_cn)

AI总结 本文提出VarTODD方法，通过分离代数重写系统与搜索策略，利用LLM引导的进化算法优化T计数，提升量子电路编译效率。

Comments 13 pages, 5 figures, 2 tables, 2 listings

Second-Order Response Laws for LLM Judges: Debiased Estimation of Prompt Instability

LLM 评判者的二阶响应定律：提示不稳定性的无偏估计

Pengbin Feng, Chunlei Meng, Daozheng Qu, Zhilin Zhang, Haoran Liu, Jiekai Wu

专题命中 评测与基准 ：LLM(title,title_cn)

AI总结 该研究针对 LLM 评判者，推导了提示不稳定性的无偏估计量，分离提示内与跨提示差异，经实验验证校正估计更准确，可单独估计提示鲁棒性。

From Survey Personas to LLM Agents: A Generative Agent-based Simulation of Mobility Policy Preference Dynamics

从调查画像到大语言模型智能体：基于生成式智能体的流动性政策偏好动态模拟

Ali Torkayesh, Julia Offermann, Regina Gimpel, Linda Engelmann, Katrin Arning, Martina Ziefle, Sandra Venghaus

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 本研究提出基于调查的生成式智能体建模框架，将德国514名调查受访者转化为LLM智能体，模拟公众对淘汰新型内燃机汽车的政策偏好动态，以助力行为实验。

Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety

谁测试测试者？LLM代理工具调用安全的系统枚举与覆盖审计

Xuan Chen, Lu Yan, Ruqi Zhang, Xiangyu Zhang

专题命中 评测与基准 ：LLM(title,title_cn)；large language model(abstract)；language model(abstract)

AI总结 本文提出SafeAudit框架，通过系统生成测试用例和规则阻力指标，发现现有测试未覆盖的20%不安全行为，揭示当前安全评估的不足。

MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

MiDashengLM-Gen：基于大语言模型驱动的自回归流匹配的统一音频场景生成

Xingwei Sun, Heinrich Dinkel, Gang Li, Jiahao Mei, Yadong Niu, Zerui Han, Yuepeng Jiang, Jiahao Zhou, Lichun Fan, Jian Luan

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 MiDashengLM-Gen是首个端到端训练的通用文本到音频生成模型，通过结合LLM与每token条件流匹配，在Seed-TTS等基准上大幅提升语音可懂度，且支持多语言场景。

ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral

ConfTriage：用于肺结节恶性程度分级的校准感知大语言模型分级框架，结合选择性专家模型转诊机制

Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin, Hasan Kurban

机构 * Texas A&M University（德克萨斯农工大学） ； Hamad Bin Khalifa University（哈马德·本·哈利法大学） 纠错

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 本研究提出ConfTriage框架，以校准感知LLM为核心结合专家DL后备模型，通过肺结节属性的自然语言表述实现分级，在LIDC-IDRI数据集上取得88.22%的F1分数与0.92的AUC，可高效处理多数病例并仅转诊不确定病例，为医疗决策支持提供了新路径。

Comments 14 pages, 8 figures. Currently under review

SurroPilot: An LLM-Assisted Platform for Heterogeneous Surrogate Endpoint Evaluation in Clinical Trials

SurroPilot：用于临床试验中异质性替代终点评估的大语言模型辅助平台

Xingyu Li, Peng Wei

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 SurroPilot是集成LLM与异质性中介框架的辅助平台，可通过自然语言交互完成临床试验异质性替代终点评估全流程，降低了相关方法的应用技术门槛。

Comments Submitted for journal

BioMedJImpact: A Comprehensive Dataset and LLM Pipeline for AI Engagement and Scientific Impact Analysis of Biomedical Journals

BioMedJImpact：用于生物医学期刊AI参与度及科学影响力分析的综合数据集与大语言模型（LLM）流程

Ruiyu Wang, Yuzhang Xie, Xiao Hu, Carl Yang, Jiaying Lu

专题命中 评测与基准 ：LLM(title,title_cn)

AI总结 本研究构建了生物医学领域的BioMedJImpact数据集，通过三阶段LLM流程提取AI参与度，分析发现作者规模与期刊影响力正相关、AI参与度仅在2019年子集与影响因子正相关，验证了LLM流程的有效性。

SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References

SONAR：面向LLM使用者的任务感知型代码摘要评估框架（无需参考摘要）

Simantika Bhattacharjee Dristi, Matthew B. Dwyer

专题命中 评测与基准 ：LLM(title,title_cn)

AI总结 该研究提出无需参考摘要的SONAR框架，从四个维度评估代码摘要，发现正确性、抽象性对LLM性能影响显著，简洁性、流畅性影响微弱，还明确了不同LLM在各维度的优劣。

Evaluating MFU as a Proxy for GPU Power for Energy-Aware Simulation of LLM Training

评估MFU作为GPU功耗代理用于LLM训练的能效感知仿真

Niklas Enskat, Philipp Wiesner

专题命中 评测与基准 ：LLM(title,title_cn)

AI总结 本研究测试MFU能否作为LLM训练的GPU功耗预测器，通过近3000次单设备训练基准测试，发现计算密集型 workload 下基于MFU的线性模型适用，特定参数拟合可大幅降低误差。

Comments Accepted at MASCOTS 2026

CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation

CodeAssay：带有审计基准真值的多指标大语言模型代码生成基准

Shahbaz Siddeeq, Muhammad Waseem, Umar Subhan Malhi, Pekka Abrahamsson

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 本文提出CodeAssay基准，涵盖185个Python任务，结合多类测试与度量，经审计后发现模型正确性标签有9.0%变化，且验证了多类LLM代码生成的评估特性。

Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking

位置偏差破坏基于大语言模型的列表式重排序中的偏好一致性

Ethan Bito, Yongli Ren, Estrid He

专题命中 评测与基准 ：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结 本研究揭示基于LLM的列表式重排序存在位置偏差，引入多维度评估框架验证其会破坏偏好一致性，且均衡位置曝光无法修复该问题。

Comments Accepted at RecSys 2026

Emergence of Biased Consensus in Multi-Agent LLM Debates

多智能体LLM辩论中偏向性共识的涌现

Maya Okawa

专题命中 评测与基准 ：LLM(title,title_cn)

AI总结 该研究针对多智能体LLM辩论中集体偏向性共识的涌现问题，提出物理启发的社会动力学分析框架，经实验验证其相变规律，发现智能体异质性可抑制该现象，且见解可推广至投资等现实决策任务。

Comments Accepted at the 43rd International Conference on Machine Learning (ICML 2026). 23 pages, 12 figures

Everyone Conforms, No One Believes: Pluralistic Ignorance in LLM Agent Populations

人人遵从，无人认同：LLM智能体群体中的多元无知

Yashwanth YS

专题命中 评测与基准 ：LLM(title,title_cn)

AI总结 该研究证实LLM智能体群体中会出现多元无知，构建了多场景基准评估8种模型，发现遵从率高且级联成功率低，提示模型选择会影响模拟结果，LLM模拟或高估社会规范稳定性。

Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing

顺应阻力：偏好优化的大型语言模型（LLM）咨询师可在动机访谈中权衡目标坚持性与关系调谐

Weiying Chen, Junlong Shen, Zhexuan Tang

专题命中 评测与基准 ：LLM(title,title_cn)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 该研究基于动机访谈准则构建双轴评估框架，通过偏好优化训练LLM咨询师，发现惩罚对抗会降低目标坚持性，惩罚妥协无显著效果，仅提示控制可提升关系调谐且无目标坚持性代价。