语言大模型 / LLM - arXivDaily 专题

2304.08865 2023-04-19 cs.CL cs.LG 84%

Romanization-based Large-scale Adaptation of Multilingual Language Models

Sukannya Purkayastha, Sebastian Ruder, Jonas Pfeiffer, Iryna Gurevych, Ivan Vulić

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 9 pages, 5 figures

2202.07206 2023-03-17 cs.CL cs.LG 84%

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Yasaman Razeghi, Robert L. Logan, Matt Gardner, Sameer Singh

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2110.03501 2023-03-15 stat.ML cs.CL cs.LG 84%

Pretrained Language Models are Symbolic Mathematics Solvers too!

Kimia Noorbakhsh, Modar Sulaiman, Mahdi Sharifi, Kallol Roy, Pooyan Jamshidi

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2303.02995 2023-03-07 cs.CV cs.CL cs.LG 84%

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, Yongfeng Zhang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at ICLR 2023

2202.07646 2023-03-07 cs.LG cs.CL 84%

Quantifying Memorization Across Neural Language Models

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2206.07699 2023-02-20 cs.CV cs.CL cs.LG 84%

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, Jiawei Wang

专题命中预训练与数据：language model(title,abstract)；foundation model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2023

2210.01296 2023-02-17 cs.CL cs.AI 84%

Recitation-Augmented Language Models

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, Denny Zhou

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICLR 2023. v2 adds the Codex results

2302.07735 2023-02-16 cs.CL cs.AI cs.CR 84%

Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Ali Al-Kaswan, Maliheh Izadi, Arie van Deursen

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2212.14034 2022-12-29 cs.CL cs.LG 84%

Cramming: Training a Language Model on a Single GPU in One Day

Jonas Geiping, Tom Goldstein

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 22 pages, we provide code at https://github.com/JonasGeiping/cramming

2208.06049 2022-12-21 cs.CV cs.CL cs.LG 84%

MILAN: Masked Image Pretraining on Language Assisted Representation

Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, Sun-Yuan Kung

专题命中预训练与数据：pretraining(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments add new experiments and improved results. provide repo link

2202.06539 2022-12-21 cs.CR cs.CL cs.LG 84%

Deduplicating Training Data Mitigates Privacy Risks in Language Models

Nikhil Kandpal, Eric Wallace, Colin Raffel

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments ICML 2022 Camera Ready Version

2211.17201 2022-12-01 cs.CL cs.LG math.OC 84%

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Rui Pan, Shizhe Diao, Jianlin Chen, Tong Zhang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2112.10668 2022-11-11 cs.CL cs.AI 84%

Few-shot Learning with Multilingual Language Models

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

专题命中预训练与数据：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2022; 34 pages

2201.06910 2022-11-01 cs.LG cs.CL 84%

ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, Zhilin Yang

专题命中预训练与数据：pretraining(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments 18 pages

2111.10962 2022-10-20 cs.CL cs.AI 84%

Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

Linlin Liu, Xin Li, Ruidan He, Lidong Bing, Shafiq Joty, Luo Si

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2022

2208.02812 2022-10-13 cs.CV cs.AI cs.LG 84%

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, Jiwen Lu

专题命中预训练与数据：prompting(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Accepted to NeurIPS 2022, project page: https://p2p.ivg-research.xyz

2202.04538 2022-10-13 cs.CL cs.LG 84%

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

Yu Meng, Jiaxin Huang, Yu Zhang, Jiawei Han

专题命中预训练与数据：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2022. (Code: https://github.com/yumeng5/SuperGen)

2209.01712 2022-09-07 cs.LG cs.AI q-bio.BM 84%

ChemBERTa-2: Towards Chemical Foundation Models

Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar

专题命中预训练与数据：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments ELLIS Machine Learning for Molecule Discovery Workshop

2111.04130 2022-07-25 cs.CL cs.LG 84%

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

Xingcheng Yao, Yanan Zheng, Xiaocong Yang, Zhilin Yang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 14 pages, 5 figures

Journal ref Proceedings of the 39th International Conference on Machine Learning, PMLR 162:25438-25451, 2022

2207.10245 2022-07-22 cs.CL cs.AI 84%

The Birth of Bias: A case study on the evolution of gender bias in an English language model

Oskar van der Wal, Jaap Jumelet, Katrin Schulz, Willem Zuidema

专题命中预训练与数据：language model(title,abstract)；small language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at the 4th Workshop on Gender Bias in Natural Language Processing (NAACL, 2022)

2206.14355 2022-06-30 cs.CV cs.CL cs.LG 84%

EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering

Violetta Shevchenko, Ehsan Abbasnejad, Anthony Dick, Anton van den Hengel, Damien Teney

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2205.01863 2022-06-27 cs.CL cs.CR cs.LG 84%

Provably Confidential Language Modelling

Xuandong Zhao, Lei Li, Yu-Xiang Wang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments NAACL 2022

2204.05939 2022-04-13 cs.CL cs.AI 84%

Mining Logical Event Schemas From Pre-Trained Language Models

Lane Lawley, Lenhart Schubert

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments To appear at ACL SRW 2022

2111.04909 2022-04-12 cs.CL cs.AI 84%

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Dezhou Shen

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments 12 pages, 4 tables

2204.03243 2022-04-08 cs.CL cs.LG 84%

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2022. (Code and Models: https://github.com/microsoft/AMOS)

2203.13240 2022-03-25 cs.CL cs.LG 84%

Token Dropping for Efficient BERT Pretraining

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, Denny Zhou

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ACL 2022

2110.04541 2022-03-22 cs.CL cs.LG 84%

The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, Amnon Shashua

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2110.09456 2021-11-02 cs.CL cs.AI 84%

NormFormer: Improved Transformer Pretraining with Extra Normalization

Sam Shleifer, Jason Weston, Myle Ott

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

2102.04130 2021-10-29 cs.CL cs.AI 84%

Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

Hannah Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frederic A. Dreyer, Aleksandar Shtedritski, Yuki M. Asano

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NeurIPS 2021. Code and data at https://github.com/oxai/intersectional_gpt2

2110.10319 2021-10-22 cs.CL cs.CY cs.IR cs.LG 84%

LMSOC: An Approach for Socially Sensitive Pretraining

Vivek Kulkarni, Shubhanshu Mishra, Aria Haghighi

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Camera ready version. Accepted to EMNLP 2021 Findings. Code for reproducing the experiments can be found at: https://github.com/twitter-research/lmsoc

1. 预训练与数据 12379 篇

Romanization-based Large-scale Adaptation of Multilingual Language Models

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Pretrained Language Models are Symbolic Mathematics Solvers too!

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Quantifying Memorization Across Neural Language Models

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Recitation-Augmented Language Models

Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Cramming: Training a Language Model on a Single GPU in One Day

MILAN: Masked Image Pretraining on Language Assisted Representation

Deduplicating Training Data Mitigates Privacy Risks in Language Models

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Few-shot Learning with Multilingual Language Models

ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

ChemBERTa-2: Towards Chemical Foundation Models

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

The Birth of Bias: A case study on the evolution of gender bias in an English language model

EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering

Provably Confidential Language Modelling

Mining Logical Event Schemas From Pre-Trained Language Models

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

Token Dropping for Efficient BERT Pretraining

The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

NormFormer: Improved Transformer Pretraining with Extra Normalization

Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

LMSOC: An Approach for Socially Sensitive Pretraining