语言大模型 / LLM - arXivDaily 专题

2301.00068 2024-02-26 cs.CL cs.AI 84%

Inconsistencies in Masked Language Models

Tom Young, Yunan Chen, Yang You

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

2402.09305 2024-02-15 cs.LG cs.AI 84%

Embracing the black box: Heading towards foundation models for causal discovery from time series data

Gideon Stein, Maha Shadaydeh, Joachim Denzler

专题命中预训练与数据：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments AAAI Workshop (AI4TS) 2024

2305.15096 2024-02-13 cs.CL cs.AI 84%

Dynamic Masking Rate Schedules for MLM Pretraining

Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, Matthew L. Leavitt

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

2303.13496 2024-01-26 cs.CV cs.AI cs.LG 84%

The effectiveness of MAE pre-pretraining for billion-scale pretraining

Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, Rohit Girdhar, Ishan Misra

专题命中预训练与数据：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Comments ICCV 2023. Models available at https://github.com/facebookresearch/maws/

2401.10278 2024-01-22 eess.SP cs.AI cs.LG cs.MM q-bio.NC 84%

EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model

Yuqi Chen, Kan Ren, Kaitao Song, Yansen Wang, Yifan Wang, Dongsheng Li, Lili Qiu

专题命中预训练与数据：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments A preprint version of an ongoing work

2312.17482 2024-01-17 cs.CL cs.LG 84%

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, Jonathan Frankle

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 10 pages, 4 figures in main text. 25 pages total

Journal ref NeurIPS 2023

2307.01163 2024-01-15 cs.CL cs.LG cs.NE 84%

Improving Language Plasticity via Pretraining with Active Forgetting

Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebastian Riedel, Mikel Artetxe

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2023 Final Version

2303.03919 2023-12-15 cs.LG cs.CL 84%

Data Portraits: Recording Foundation Model Training Data

Marc Marone, Benjamin Van Durme

专题命中预训练与数据：foundation model(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2023 Datasets and Benchmarks

2302.03169 2023-11-21 cs.CL cs.LG 84%

Data Selection for Language Models via Importance Resampling

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2023

2311.03250 2023-11-07 cs.CL cs.AI 84%

Instructed Language Models with Retrievers Are Powerful Entity Linkers

Zilin Xiao, Ming Gong, Jie Wu, Xingyao Zhang, Linjun Shou, Jian Pei, Daxin Jiang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2023 Main

2311.02597 2023-11-07 cs.AI cs.CL 84%

FloodBrain: Flood Disaster Reporting by Web-based Retrieval Augmented Generation with an LLM

Grace Colverd, Paul Darm, Leonard Silverberg, Noah Kasmanoff

专题命中预训练与数据：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Version is the one submitted to Artificial Intelligence for Humanitarian Assistance and Disaster Response Workshop @Neurips2023. All authors contributed equally to this work

2303.12513 2023-11-07 cs.CV cs.CL cs.LG 84%

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

Morris Alper, Michael Fiman, Hadar Averbuch-Elor

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted to CVPR 2023. Project webpage: https://isbertblind.github.io/

2310.19961 2023-11-01 cs.LG cs.AI 84%

ExPT: Synthetic Pretraining for Few-Shot Experimental Design

Tung Nguyen, Sudhanshu Agrawal, Aditya Grover

专题命中预训练与数据：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Comments 2023 Conference on Neural Information Processing Systems (NeurIPS)

2310.17530 2023-10-27 cs.CV cs.CL cs.LG 84%

Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models

Laura Cabello, Emanuele Bugliarello, Stephanie Brandl, Desmond Elliott

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments To appear in EMNLP 2024

2310.14369 2023-10-24 cs.LG cs.AI 84%

MoPe: Model Perturbation-based Privacy Attacks on Language Models

Marvin Li, Jason Wang, Jeffrey Wang, Seth Neel

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.AI、cs.LG

2310.03477 2023-10-06 cs.CL cs.AI 84%

Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

François Remy, Pieter Delobelle, Bettina Berendt, Kris Demuynck, Thomas Demeester

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments As first reviewed at TACL

2309.09380 2023-09-19 cs.CL cs.LG 84%

Mitigating Shortcuts in Language Models with Soft Label Encoding

Zirui He, Huiqi Deng, Haiyan Zhao, Ninghao Liu, Mengnan Du

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2308.12270 2023-08-24 cs.LG cs.AI 84%

Language Reward Modulation for Pretraining Reinforcement Learning

Ademi Adeniji, Amber Xie, Carmelo Sferrazza, Younggyo Seo, Stephen James, Pieter Abbeel

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Code available at https://github.com/ademiadeniji/lamp

2308.00624 2023-08-02 cs.CL cs.AI 84%

JIANG: Chinese Open Foundation Language Model

Qinhua Duan, Wenchao Gu, Yujia Chen, Wenxin Mao, Zewen Tian, Hui Cao

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2307.14430 2023-08-01 cs.CL cs.LG 84%

Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, Christopher Ré

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2307.04401 2023-07-11 cs.CL cs.AI 84%

Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation

Zhexin Zhang, Jiaxin Wen, Minlie Huang

专题命中预训练与数据：prompting(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments ACL 2023 Long Paper (Main Conference)

2212.10785 2023-05-30 cs.CL cs.AI 84%

SERENGETI: Massively Multilingual Language Models for Africa

Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Alcides Alcoba Inciarte

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments To appear in Findings of ACL 2023

2305.15507 2023-05-26 cs.CL cs.AI 84%

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Antonio Valerio Miceli-Barone, Fazl Barez, Ioannis Konstas, Shay B. Cohen

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments 17 pages, 5 figure, ACL 2023

2305.12281 2023-05-23 cs.CL cs.LG 84%

Lifelong Language Pretraining with Distribution-Specialized Experts

Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, Claire Cu

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICML 2023

2305.10992 2023-05-19 cs.CL cs.AI 84%

How does the task complexity of masked pretraining objectives affect downstream performance?

Atsuki Yamaguchi, Hiroaki Ozaki, Terufumi Morishita, Gaku Morio, Yasuhiro Sogawa

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ACL 2023 Findings

2209.10372 2023-05-17 cs.CL cs.AI 84%

WeLM: A Well-Read Pre-trained Language Model for Chinese

Hui Su, Xiao Zhou, Houjin Yu, Xiaoyu Shen, Yuwen Chen, Zilin Zhu, Yang Yu, Jie Zhou

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2211.00151 2023-05-09 cs.CL cs.LG 84%

A Close Look into the Calibration of Pre-trained Language Models

Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, Heng Ji

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ACL 2023 main conference. Code is available at: https://github.com/lifan-yuan/PLMCalibration

2305.01550 2023-05-03 cs.CL cs.CR cs.LG 84%

Mitigating Approximate Memorization in Language Models via Dissimilarity Learned Policy

Aly M. Kassem

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2304.14275 2023-04-28 cs.CL cs.LG 84%

What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files

Peter Meltzer, Joseph G. Lambourne, Daniele Grandi

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2304.10946 2023-04-24 cs.CL cs.LG q-bio.BM 84%

CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models

Tianhao Li, Sandesh Shetty, Advaith Kamath, Ajay Jaiswal, Xianqian Jiang, Ying Ding, Yejin Kim

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.LG

1. 预训练与数据 12379 篇

Inconsistencies in Masked Language Models

Embracing the black box: Heading towards foundation models for causal discovery from time series data

Dynamic Masking Rate Schedules for MLM Pretraining

The effectiveness of MAE pre-pretraining for billion-scale pretraining

EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

Improving Language Plasticity via Pretraining with Active Forgetting

Data Portraits: Recording Foundation Model Training Data

Data Selection for Language Models via Importance Resampling

Instructed Language Models with Retrievers Are Powerful Entity Linkers

FloodBrain: Flood Disaster Reporting by Web-based Retrieval Augmented Generation with an LLM

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

ExPT: Synthetic Pretraining for Few-Shot Experimental Design

Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models

MoPe: Model Perturbation-based Privacy Attacks on Language Models

Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

Mitigating Shortcuts in Language Models with Soft Label Encoding

Language Reward Modulation for Pretraining Reinforcement Learning

JIANG: Chinese Open Foundation Language Model

Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation

SERENGETI: Massively Multilingual Language Models for Africa

The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python

Lifelong Language Pretraining with Distribution-Specialized Experts

How does the task complexity of masked pretraining objectives affect downstream performance?

WeLM: A Well-Read Pre-trained Language Model for Chinese

A Close Look into the Calibration of Pre-trained Language Models

Mitigating Approximate Memorization in Language Models via Dissimilarity Learned Policy

What's in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models through User-Provided Names in CAD Files

CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Models