Open-World Evaluations for Measuring Frontier AI Capabilities
面向前沿AI能力的开放世界评估
Sayash Kapoor, Peter Kirgis, Andrew Schwartz, Stephan Rabanser, J. J. Allaire, Rishi Bommasani, Harry Coppock, Magda Dubois, Gillian K Hadfield, Andrew B. Hall, Sara Hooker, Seth Lazar, Steve Newman, Dimitris Papailiopoulos, Shoshannah Tekofsky, Helen Toner, Cozmin Ududec, Arvind Narayanan
机构
*
Princeton University(普林斯顿大学)
;
Cornflower Labs(Cornflower实验室)
;
Meridian Labs(Meridian实验室)
;
Stanford University(斯坦福大学)
;
UK AI Security Institute(英国人工智能安全研究所)
;
Johns Hopkins University(约翰霍普金斯大学)
;
Adaption Labs(Adaption实验室)
;
Australian National University(澳大利亚国立大学)
;
Golden Gate Institute for AI(金门人工智能研究所)
;
UW Madison(威斯康星大学麦迪逊分校)
;
Microsoft Research(微软研究院)
;
AI Digest(AI摘要)
;
Georgetown University (CSET)(乔治城大学(CSET))
EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data
EgoBabyVLM:基于自然主义第一人称视频数据的跨模态学习基准测试
Dongyan Lin, Phillip Rust, Angel Villar Corrales, Alvin W. M. Tan, Mahi Luthra, Charles-Éric Saint-James, Rashel Moritz, Sheila Krogh-Jespersen, Vanessa Stark, Surya Parimi, Jiayi Shen, Youssef Benchekroun, Yosuke Higuchi, Martin Gleize, Tom Fizycki, Nicolas Hamilakis, Manel Khentout, Sho Tsuji, Balázs Kégl, Juan Pino, Michael C. Frank, Emmanuel Dupoux
机构
*
Meta Superintelligence Labs(Meta超智能实验室)
;
Stanford University(斯坦福大学)
;
Meta Reality Labs(Meta现实实验室)
;
The University of Tokyo(东京大学)
AI总结
研究探讨了儿童如何从有限的视觉-语言输入中获得语言 grounding 的鲁棒性,提出了 EgoBabyVLM 挑战,推动模型在自然主义数据中实现 grounded language learning。
Learning Interpretable Point-Based Clinical Risk Scores via Direct Optimization
通过直接优化学习可解释的基于点的临床风险评分
Ying Cui, Albert M Li, Vivek Charu, Yeon-Mi Hwang, Tina Hernandez-Boussard, Lu Tian
机构
*
Department of Biomedical Data Science, Stanford University(斯坦福大学生物医学数据科学系)
;
Decatur High School(德凯高中)
;
Department of Pathology, Stanford University School of Medicine(斯坦福大学医学院病理学系)
;
Division of Computational Medicine, Department of Medicine, Stanford University(斯坦福大学医学系计算医学分会)
机构
*
Université de Montréal(蒙特利尔大学)
;
The University of Sydney(悉尼大学)
;
Fudan University(复旦大学)
;
Yale University(耶鲁大学)
;
DeepWisdom
;
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
;
Amazon Science(亚马逊科学)
;
Stanford University(斯坦福大学)
;
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
;
Mila - Quebec AI Institute(蒙特利尔人工智能研究所)
CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
CHI-Bench: 能否让AI代理自动化端到端、长周期、政策丰富的医疗工作流程?
Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps MD, Zeyu Tang, Xiangchen Song, Lingjing Kong, Fan Feng, Tianyi Zeng, Zhiwei Liu, Zixian Ma, Hang Jiang, Fangli Geng, Yuan Yuan, Chenyu You, Qingsong Wen, Hua Wei, Yanjie Fu, Yue Zhao, Carl Yang, Biwei Huang, Kun Zhang, Caiming Xiong, Sanmi Koyejo, Eric P. Xing, Philip S. Yu, Weiran Yao
机构
*
Johns Hopkins Medicine(约翰霍普金斯医学中心)
;
Wellstar Health System(Wellstar健康系统)
;
Stanford University(斯坦福大学)
;
CMU(卡内基梅隆大学)
;
UCSD(加州大学圣地亚哥分校)
;
Yale School of Medicine(耶鲁医学院)
;
Salesforce AI Research(Salesforce人工智能研究)
;
University of Washington(华盛顿大学)
;
Northeastern University(东北大学)
;
Brown University(布朗大学)
;
Boston College(波士顿学院)
;
Stony Brook University(史泰森布里克大学)
;
University of Oxford(牛津大学)
;
Arizona State University(亚利桑那州立大学)
;
University of Southern California(南加州大学)
;
Emory University(埃默里大学)
;
MBZUAI
;
Recursive Superintelligence(递归超级智能)
;
University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
大规模监控AI修改内容:ChatGPT对AI会议同行评审影响的案例研究
Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, James Y. Zou
机构
*
Department of Computer Science, Stanford University(斯坦福大学计算机科学系)
;
Machine Learning Department, NEC Labs America(NEC美国实验室机器学习部门)
;
Department of Biomedical Data Science, Stanford University(斯坦福大学生物医学数据科学系)
;
Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系)
;
Graduate School of Education, Stanford University(斯坦福大学教育研究生院)
;
Department of Sociology, Stanford University(斯坦福大学社会学系)
;
Graduate School of Business, Stanford University(斯坦福大学商学院)
;
Department of Management Science and Engineering, Stanford University(斯坦福大学管理科学与工程系)
;
Department of Computer Science, UC Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
机构
*
Computer Vision Institute, School of Computer Science and Software Engineering, Shenzhen University(计算机视觉研究院,计算机科学与软件工程学院,深圳大学)
;
Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University(广东省智能信息处理重点实验室,深圳大学)
;
School of Computer Science, University of Nottingham Ningbo China(Nottingham Ningbo 中国计算机科学学院)
;
Department of Electrical and Computer Engineering, National University of Singapore(电子与计算机工程系,新加坡国立大学)
;
Department of Radiation Oncology, Stanford University(放射肿瘤科,斯坦福大学)
;
Sun Yat-sen University(中山大学)
;
School of Computer Science, University of Nottingham(计算机科学学院,Nottingham大学)
Leveraging Speech to Identify Signatures of Insight and Transfer in Problem Solving
利用语音识别洞察和迁移的特征
Linas Nasvytis, Judith E. Fan
机构
*
Department of Psychology, Stanford University(斯坦福大学心理学系)
;
Graduate School of Education, Stanford University(斯坦福大学教育研究生院)
;
Department of Computer Science, Stanford University(斯坦福大学计算机科学系)
Katherine M. Collins, Cedegao E. Zhang, Graham Todd, Lance Ying, Mauricio Barba da Costa, Ryan Liu, Prafull Sharma, Adrian Weller, Ionatan Kuperwajs, Lionel Wong, Joshua B. Tenenbaum, Thomas L. Griffiths
机构
*
University of Cambridge(剑桥大学)
;
MIT(麻省理工学院)
;
Princeton University(普林斯顿大学)
;
NYU(纽约大学)
;
Harvard University(哈佛大学)
;
Stanford University(斯坦福大学)
Prioritizing High-Consequence Biological Capabilities in Evaluations of Artificial Intelligence Models
在评估人工智能模型时优先考虑高后果生物能力
Jaspreet Pannu, Doni Bloomfield, Alex Zhu, Robert MacKnight, Gabe Gomes, Anita Cicero, Thomas V. Inglesby
机构
*
Center for Health Security, Bloomberg School of Public Health, Johns Hopkins University(健康安全中心,公共卫生学院,约翰霍普金斯大学)
;
Department of Health Policy, Stanford School of Medicine, Stanford University(健康政策系,斯坦福医学院,斯坦福大学)
;
Department of Chemical Engineering, Carnegie Mellon University(化学工程系,卡内基梅隆大学)
;
Department of Chemistry, Carnegie Mellon University(化学系,卡内基梅隆大学)
;
Wilton E. Scott Institute for Energy Innovation, Carnegie Mellon University(威尔顿·E·斯科特能源创新研究所,卡内基梅隆大学)
WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform
WorldArena 2.0: 扩展模态、功能和平台的具身世界模型基准测试
Yu Shang, Yinzhou Tang, Yiding Ma, Zhuohang Li, Lei Jin, Weikang Su, Xin Jin, Zhaolu Wang, Ziyou Wang, Xin Zhang, Haisheng Su, Weizhen He, Wei Wu, Haoyi Duan, Gordon Wetzstein, Xihui Liu, Dhruv Shah, Zhaoxiang Zhang, Zhibo Chen, Jun Zhu, Yonghong Tian, Tat-Seng Chua, Wenwu Zhu, Chen Gao, Yong Li
机构
*
Tsinghua University(清华大学)
;
Shanghai Jiao Tong University(上海交通大学)
;
Zhejiang University(浙江大学)
;
Stanford University(斯坦福大学)
;
The University of Hong Kong(香港大学)
;
Princeton University(普林斯顿大学)
;
Chinese Academy of Sciences(中国科学院)
;
University of Science and Technology of China(中国科学技术大学)
;
Peking University(北京大学)
;
National University of Singapore(新加坡国立大学)
Simple Approximation and Derivative Free Inference-Time Scaling for Diffusion Models via Sequential Monte Carlo on Path Measures
通过路径测度的序列蒙特卡洛实现扩散模型的简单近似与无导数推理时间缩放
Chenyang Wang, Weizhong Wang, Yinuo Ren, Jose Blanchet, Yiping Lu
机构
*
School of Mathematical Sciences, Peking University, Beijing, China
;
School of Mathematical Sciences, Fudan University, Shanghai, China
;
Department of Industrial Engineering \& Management Sciences, Northwestern University, Evanston, IL, United States
;
Institute for Computational \& Mathematical Engineering, Stanford University, Stanford, CA, United States
;
Management Science \& Engineering, Stanford University, Stanford, CA, United States
机构
*
University of Texas Austin(德克萨斯大学奥斯汀分校)
;
California Institute of Technology(加州理工学院)
;
Carnegie Mellon University(卡内基梅隆大学)
;
Stanford University(斯坦福大学)
;
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
;
Microsoft Research(微软研究院)
;
Northwestern University(西北大学)
;
University of Cambridge(剑桥大学)
Zhi Li, Songkun Yan, Jie Cao, Mofan Zhang, Anjiang Wei, Jinwoong Yoo, Yang Hong
机构
*
Civil, Environmental, and Architectural Engineering, University of Colorado Boulder(科罗拉多大学波尔德分校土木、环境与建筑工程系)
;
Civil Engineering and Environmental Sciences, University of Oklahoma(俄克拉荷马大学土木工程与环境科学系)
;
Department of Computer Science, University of Oklahoma(俄克拉荷马大学计算机科学系)
;
Civil and Environmental Engineering, Stanford University(斯坦福大学土木与环境工程系)
;
Department of Computer Science, Stanford University(斯坦福大学计算机科学系)
;
NASA Goddard Space Flight Center(美国国家航空航天局戈达德空间飞行中心)
机构
*
Dept. of Computer Science and Engg., IIT Kharagpur(印度Kharagpur理工学院计算机科学与工程系)
;
Dept. of Medicine (Biomedical Informatics), Stanford University(斯坦福大学医学院(生物医学信息学))
机构
*
University of North Carolina at Chapel Hill(北卡罗来纳大学切里波因特分校)
;
Stanford University(斯坦福大学)
;
University of California, Berkeley(加州大学伯克利分校)
;
University of Southern California(南加州大学)
;
University of California, Santa Cruz(加州大学圣克鲁兹分校)