Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
迷失在标记化中:图标记化在Transformer中的基本权衡
Maya Bechler-Speicher, Gilad Yehudai, Gil Harari, Clayton Sanford, Amir Globerson, Joan Bruna
机构
*
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院)
;
John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院)
;
Google Research(谷歌研究)
;
Tel-Aviv University(特拉维夫大学)
Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
深度网络中的谱动力学:特征学习、异常值逃逸和学习率转移
Clarissa Lauditi, Cengiz Pehlevan, Blake Bordelon
机构
*
John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰A·保罗森工程与应用科学学院)
;
Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究学院)
;
Center for Mathematical Sciences and Applications, Harvard University(哈佛大学数学科学中心)
;
Oden Institute for Computational Engineering and Sciences & Dept. of Neuroscience, UT Austin(得克萨斯大学奥斯汀分校奥登计算工程与科学学院及神经科学系)
Hyperparameter Transfer with Mixture-of-Expert Layers
通过专家混合层进行超参数迁移
Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin
机构
*
Operations Research
;
Financial Engineering, Princeton University, Princeton, NJ, USA
;
Center of Mathematical Sciences
;
Applications, Harvard University, Cambridge, MA, USA
;
John A. Paulson School of Engineering
;
Applied Sciences, Center for Brain Science, Kempner Institute for the Study of Natural
;
Artificial Intelligence, Harvard University, Cambridge, MA, USA
机构
*
New York University Abu Dhabi(纽约大学阿布扎克校区)
;
Nanyang Technological University(南洋理工大学)
;
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
;
Harvard University(哈佛大学)
;
Zhejiang University(浙江大学)
;
University of Electronic Science and Technology of China(电子科技大学)
;
Beijing University of Technology(北京理工大学)
;
The Hong Kong Polytechnic University(香港理工大学)
Yasha Ektefaie, Leo Cui, Shrey Jain, Marinka Zitnik, Pardis Sabeti
机构
*
Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard(埃里克和wendy Schmidt中心,MIT和哈佛大学Broad研究所)
;
Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系)
;
Centennial High School(Centennial高中)
;
Infectious Disease and Microbiome Program, Broad Institute of MIT and Harvard(传染病与微生物组计划,MIT和哈佛大学Broad研究所)
An AI system to help scientists write expert-level empirical software
一种帮助科学家编写专家级经验软件的AI系统
Eser Aygün, Anastasiya Belyaeva, Gheorghe Comanici, Marc Coram, Hao Cui, Jake Garrison, Renee Johnston Anton Kast, Cory Y. McLean, Peter Norgaard, Zahra Shamsi, David Smalling, James Thompson, Subhashini Venugopalan, Brian P. Williams, Chujun He, Sarah Martinson, Martyna Plomecka, Lai Wei, Yuchen Zhou, Qian-Ze Zhu, Matthew Abraham, Erica Brand, Anna Bulanova, Jeffrey A. Cardille, Chris Co, Scott Ellsworth, Grace Joseph, Malcolm Kane, Ryan Krueger, Johan Kartiwa, Dan Liebling, Jan-Matthis Lueckmann, Paul Raccuglia, Xuefei, Wang, Katherine Chou, James Manyika, Yossi Matias, John C. Platt, Lizzie Dorfman, Shibl Mourad, Michael P. Brenner
机构
*
Google DeepMind(谷歌DeepMind)
;
Google Research(谷歌研究)
;
Google Platforms and Devices(谷歌平台与设备)
;
Massachusetts Institute of Technology(麻省理工学院)
;
School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院)
AI总结
本文提出Empirical Research Assistance (ERA)系统,利用大型语言模型和树搜索技术,自动创建高质量的科学软件,以加速计算实验的开发,从而提高科研效率。