Lessons from the Trenches on Reproducible Evaluation of Language Models
关于语言模型可重复评估的前线经验教训
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, Andy Zou
机构
*
MBZUAI
;
IIIT Hyderabad
;
EleutherAI
;
HiTZ Center - Ixa, UPV/EHU
;
Ivy Natal
;
University of Michigan
;
HubSpot
;
LibrAI
;
Kensho
;
Contextual AI
;
Brown University
;
New York University
;
Amazon
;
Yale University
;
HKUST
;
Sorbonne University
;
CMU
AI总结
本文基于开发Language Model Evaluation Harness框架的三年经验,总结了语言模型评估中面临的方法论挑战、缺乏可重复性和透明度等问题,并提供了改进评估严谨性和信心的建议。
Synthesizing Neural Network Controllers with Closed-Loop Dissipativity Guarantees
具有闭环耗散性保证的神经网络控制器综合
Neelay Junnarkar, Murat Arcak, Peter Seiler
机构
*
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley(加州大学伯克利分校电子工程与计算机科学系)
;
Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor(密歇根大学安娜堡分校电子工程与计算机科学系)
On-Device Robotic Planning: Eliminating Inference Redundancy for Efficient Decision-Making
设备端机器人规划:消除推理冗余以实现高效决策
Joonhee Lee, Hyunseung Shin, Hyunmi Kim, Pei Zhang, Jeonggil Ko
机构
*
School of Integrated Technology, College of Computing, Yonsei University(延世大学整合技术学院,计算学院)
;
Department of Hyperscale AI SoC Research Section, ETRI(ETRI超大规模AI SoC研究部)
;
EECS - Electrical and Computer Engineering, University of Michigan(密歇根大学电气与计算机工程学院)
机构
*
University of California, San Diego(加州大学圣地亚哥分校)
;
University of Michigan, Ann Arbor(密歇根大学安娜堡分校)
;
The Hong Kong University of Science and Technology, Guangzhou(香港科学与技术大学(广州))
;
ShanghaiTech University(上海科技大学)
;
University of North Carolina, Chapel Hill(北卡罗来纳大学教堂山分校)
机构
*
Department of Biomedical Informatics and Data Science, Yale School of Medicine(耶鲁医学院生物医学信息学与数据科学部)
;
School of Information, University of Michigan(密歇根大学信息学院)
Trajectory Constraints for Imaging Inverse Problems
成像逆问题的轨迹约束
Chaoyan Huang, Haijie Yuan, Saiprasad Ravishankar
机构
*
Department of Computational Mathematics, Science, & Engineering, Michigan State University(密歇根州立大学计算数学、科学与工程系)
;
Department of Electrical Engineering and Computer Science, University of Michigan(密歇根大学电气工程与计算机科学系)
;
Department of Biomedical Engineering, Michigan State University(密歇根州立大学生物医学工程系)
A Principled Self-Referenced Early Stopping Approach for Deep Image Prior
一种基于自引用的原则性早期停止方法用于深度图像先验
Chaoyan Huang, Cheng-Han Huang, Ismail R. Alkhouri, Rongrong Wang
机构
*
Department of Computational Mathematics, Science, & Engineering, Michigan State University(密歇根州立大学计算数学、科学与工程系)
;
Department of Electrical Engineering and Computer Science, University of Michigan(密歇根大学电气工程与计算机科学系)
;
X Computational Physics Division, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室计算物理部)
;
Michigan Institute for Computational Discovery & Engineering, University of Michigan(密歇根大学计算发现与工程研究所)
;
Mathematical Sciences, Michigan State University(密歇根州立大学数学科学系)
History-aware adaptive reduced-order models via incremental singular value decomposition
基于增量奇异值分解的历史感知自适应降阶模型
Amirpasha Hedayat, Ali Mohaghegh, Laura Balzano, Cheng Huang, Karthik Duraisamy
机构
*
Department of Aerospace Engineering(航空航天工程系)
;
University of Michigan(密歇根大学)
;
University of Kansas(堪萨斯大学)
;
Department of Electrical Engineering and Computer Science(电气工程与计算机科学系)
KT4EQG: Personalized Exercise Question Generation via Knowledge Tracing
KT4EQG: 通过知识追踪实现个性化习题生成
Xinyi Gao, Qiucheng Wu, Lu Ding, Q. Vera Liao, Kaizhi Qian, Ying Xu, Shiyu Chang, Yang Zhang
机构
*
University of California, Santa Barbara(加州大学圣巴巴拉分校)
;
University of South Alabama(南方大学)
;
University of Michigan(密歇根大学)
;
MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室)
;
Harvard University(哈佛大学)
Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction
利用大语言模型和信息论量化天体物理方法的可重建性:光谱重建的案例研究
Hsing Wen Lin, Zong-Fu Sie
机构
*
Department of Physics, University of Michigan, Ann Arbor, MI 48109, USA(密歇根大学物理系)
;
Michigan Institute for Data and AI in Society, University of Michigan, Ann Arbor, MI 48109, USA(密歇根大学数据与人工智能在社会中的研究所)
;
Independent Researcher, Taiwan(台湾独立研究员)
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Camellia: 亚洲语言中LLMs文化偏见的基准测试
Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah Masud, Tanish Patwa, Trung Thanh Tran, Zohaib Khan, Alan Ritter, Tanmoy Chakraborty, Yuki Arase, Keisuke Sakaguchi, JinYeong Bak, Wei Xu
机构
*
Georgia Institute of Technology(佐治亚理工学院)
;
Samsung R&D Institute Philippines(三星菲律宾研发院)
;
Sungkyunkwan University(成均馆大学)
;
Tohoku University(东北大学)
;
National University of Singapore(新加坡国立大学)
;
University of Copenhagen(哥本哈根大学)
;
University of Michigan(密歇根大学)
;
Indian Institute of Technology Delhi(印度理工学院德里)
;
Institute of Science Tokyo(东京科学研究院)
Aishwarya Mandyam, Shengpu Tang, Jiayu Yao, Jenna Wiens, Barbara E. Engelhardt
机构
*
Stanford University(斯坦福大学)
;
Emory University(埃默里大学)
;
Columbia University(哥伦比亚大学)
;
University of Michigan(密歇根大学)
;
The Gladstone Institutes(加利福尼亚大学旧金山分校格罗斯曼研究所)